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ABSTEACT 

The Instructional Improvement Questionnaire (HQ) is 
a student rating form designed to provide evaluative feedback to 
instructors about their teaching. The HQ nas first developed at 
Southern Illinois University at Carbondale in 1969 and revised in 
1972. The documents included here describe the development of the HQ 
and research associated vith it. Besults from the HQ may be used for 
teacher promotion and annual salary reviens, student course 
selection* and evaluating course effectiveness. Eesults are released 
only on authorization of the instructor. The present HQ form has 
four parts: a student biographic section, an instructor evaluation 
section (20 items), a course evaluation section (20 items), and an 
optional item section nhere instructors can have students respond to 
as many as 60 items prepared by individual departments or faculty. 
Students respond directly on OpScan answer sheets vhich are computer 
processed, additional research reported deals nith an effectiveness 
study of the evaluation of instruction program by the faculty in 
ninter 1973, a cross-validation of the HQ deciles from 1971 to fall 
1972, norms for required and elective courses by course level for the 
IIQ^^fall 1971, a summary of research on the relationships betveen 
student characteristics and student evaluations of instruction at 
Southern Illinois University, a description of effective college 
teaching in five disciplines as measured by student ratings, and 
evaluating instructional effectiveness nith the HQ. A copy of the 
HQ is appended. (SC) 
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The Instructional Improvement Questionnaire (I 10) is a student rating 
form designed to provide evaluative feedback to instructors about their 
teaching. The I IQ was first developed at Southern Illinois University at 
Carbondaie in !969 and revised in 1972* This article describes the develop- 
ment of the i IQ and research associated with it* 
Development 

Development of the I IQ began in 1969 with a review of the existing research 
literature and copies of many student rating forms used at universities 
throughout the United States* From this review, rating items were selected 
for inclusion in a local item pool, along with locally prepared items* The 
item pool was then reviewed by a committee composed of students, faculty, and 
measurement specialists* A trial form of the I IQ was used on a pilot basis 
In thirty courses in 1969, and students and faculty who participated in the 
pilot study were asked to react to the form* After reviewing student and 
faculty comments, a 72^item form of the I IQ was prepared and administered on 
a voluntary basis for one year* Subsequently, the I IQ was further revised 
and shortened to a 49-item form which has been in use for three years* in 
, revising the MQ, three criteria were used to eliminate items* They were 
(a) time to complete the form, (b) item variability across courses, and 
(c) an item's relationship to factors derived from a factor analysis of the 
IIQ items* A section of forced-choice items, modeled after the Purdue 
Instructor Performance Indicator (Snedeker A Reramers, i960) was eliminated 
because the faculty felt the information took too long to collect* Four 
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items were removed from the MQ because the variability of responses across 
course means was too small. These same four items also possessed low loadings 
on factors derived from the ilQ items* The items retained in the IIQ possessed 
standard deviations across course means greater than *4 on a 5-point Likert 
scale. The MQ now in use can be administered to a class of 30 students in 
approximately 10 minutes, and is composed ot items that maximally differen- 
tiate among instructors* 

Participation in the IIQ evaluation at Southern Illinois University (SlU) 
is voluntary* Approximately 2,000 courses are evaluated annually with the IIQ, 
and this represents 40 percent of the courses taught at SlU in a year. Results 
from the IIQ are used in a number of ways for faculty development and faculty 
evaluation* I IQ results may be used by faculty as a measure of teaching 
effectiveness in their p»^omotion and annual salary reviews. Students have 
used MQ data to select instructors and courses* The campus Learning Resources 
Service used MQ data as one criterion when evaluating the effectiveness of 
the courses that they develop. In all cases, a faculty member's results 
are released only to individuals and agencies upon the written authorization 
of the instructor^ 

The present IIQ form has four parts; (a) a student biographic data section, 
(b) an instructor evaluaxlon section (20 items), (c) a course evaluation sec- 
tion (20 items), and (d) an optional item section where instructors can have 
students respond to items prepared by individual departments or faculty. 
The students respond to the MQ directly on an OpScan answer sheet. The 
answer sheets are scanned and computer processed* The instructor receives 
a computerized report of the studonts' responses. Special norm tables are 
used to provide results to the instructor in terms of canpus-wtde norms and 
course type norms based on required and elective courses at each l^vel 



ERIC 



a 



(freshmen, sophonores, etc.). An instructor's results are compared to the 
normative sample of over 2,000 courses evaluated in 1974-75 by quoting the 
decile equivalent of each item mean on the computer printout. Additionally, 
subscores derived from homogeneous subsets of items, identified in a factor 
analysis of the I IQ, are compared to courses at five course levels (freshman 
through graduate) for required and elective courses* The instructor's sub- 
scores are reported as decile equivalent^ in the distributions of responses 
from various combinations of courses, based on course level and the required- 
elective nature of the course* 
Research Conducted on the MQ 

Research with the I IQ has fallen into five broad categories: (a) faculty 
reactions to the IIQ, (b) reliability of I IQ results, (c) relationship between 
class characteristics and IIQ results, and (e) a description of effective 
col lege teaching using the I IQ, 

Two studies were conducted to determfne faculty reactions to the IIQ 
(Pohlmann, l973a;Elmore, 1975)* In both surveys, 83 percent of the faculty 
members using the IIQ felt that the results represented a generally accurate 
indication of their ^^^eaching effectiveness,* Only four percent responding to 
the 1975 survey considered the results not helpful or a waste of time, and 
only eighl percent in the 1975 survey considered the information provided by 
the I IQ as not useful * 

Internal consistency and test-retest reliability studies show the IIQ 
subscales to be reasonably reliable* Internal consistency coefficients 
(Cronbach's alpha) range from *62 to .93 for the five IIQ subscales. The 
subscales were derived from a faotor analysis of the 40 IIQ items* Three- 
month test-retest correlations on the five 1 IQ subscales ranged from .67 to 
.76 (Pohlmann, 1973b). The reliability studies were conducted on samples of 
dato that were not used in the factor analysis studies* 
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Pohlmann CI975b) conducted a study to determine the relationsinp between 
course characteristics and HQ results* The results of f"hat study indicated 
that the mos1 predictive class characteristic variables were the grades 
expected by the students and the percent of students taking the course as an 
elective* These two variables were positively and moderately related to 
1 10 results, 

A study conducted by Elmore and LaPointe (1974) assessed the influence 
of faculty- sex and student sex on teacher evaluations using !Qi In 

general^, this study found that there were no differences between the mean 
ratings given male and female faculty by male and female students* A followup 
s1udy (Elmore & LaPointe* 1975) analyzed perceived teacher wannth along with 
faculty sex and student sex. The results indicated that, when students 
rated their instructors' interest and warmth, teachers who were wanner and 
seemed primarily interested in students received higher ratings on teacher 
effectiveness* 

Pohlmann ( 1975a) used the 1 IQ to examine the specific rating correlates 
of a general rating of instructional effectiveness* This study suggested 
that the effectlvi teacher was an individual who was well-prepared, organized 
in presenting material, achieved the course objectives and increased the 
stucent's appreciation of the subject matter* 

In conclusion, the Instructional Improvement Questionnaire has become a 
valuable feedback device for college instructors and for research in the field 
of teacher effectiveness* 
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AN EVALUATION OF THE EVALUATION 
OF INSTRUCTION PROGRAM 
BY FACULTY, WINTER 1973' 



John T. Pohlmann 
Southern Illinois University at Carbondale 

The effectiveness of any service such as the Evaluation of Instruction 
program depends upon its responsiveness to the needs of its users^ In 
order for a service to be responsive to user needs, channels of coinmunication 
must be opened between program personnel and program users^ To this end, 
a questionnaire was distributed to faculty members who particpated in the 
Evaluation of Instruction program during Winter Quarter, 1973* The purpose 
of the questionnaire was to allow faculty users to evaluate various aspect^} 
of the service, and provide suggestions for the improvement of the service* 
This report outlines the results obtained from that questionnaire: 

The items included in the questionnaire were directed at obtaining 

answers to the following questions: 

1* What percentage of users report satisfaction with the service 
they received? 

2* Was the computer printout understandable? 

3* Were the deciles helpful for interpreting the results of the 
evaluation? 

4* Did faculty members using the MIRROR e\aluation form find 
the comments of their students helpful? 

5* Which type of Information, the written ^mments of students 
solicited on tne MIRROR questionnaire or the computerized 
results of student responses to the Instructional Improve- 
ment Questionnaire, was more informative? 

6^ What could be done to improve the evaluation service? 

A copy of the questionnaire can be found in the Appendix^ 
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Procedures and Results 

k questionnaire was mailed to each of the 268 faculty members who 
participated in the Evaluation of Instruction Program during the Winter 
Quarter, 1973* One-hundred and nineteen usable questionnaires were 
returned to the Testing Centerp This represents a 44 percent return rate^ 

The £Lrst question asked was "Were you satisfied with the service you 
received from the Testing Center?" Ninety-five percent of the responding 
faculty answered "yes" to this question^ There were six "no" responses^ 
Two of the negative responses were directed at the questionnaire and the 
voluntary nature of the service^ One respondent felt that the questionnaire 
i^ too complicated, and the other respondent felt that a larger number of 
faculty member* s results should be reflected in the norms, The other four 
respondents who stated that they were not satisfied with the service, did 
so because of poor service from the Testing Center. Three of these instruc- 
tors failed to get their results^ Upon examining the records from the 
Winter Quarter evaluation, the source of the problem was found to be errors 
in coding the faculty members materials when they were brought to the 
Testing Center^ Either the wrong log number was assigned to a batch of materi 
or the instructor's department was coded incorrectly* One instructor said 
he was dissatisfiedwith the service because it took two weeks to get his 
results. This is, in one sense, a compliment since turn around time for the 
evaluation program last year was approximately five weeks for everybody* 
This suggests that an appreciable improvement in turn around time has been 
realized this year* 

The second question asked the instructor if the computer printout was 
understandable* Eighty-nine percent of the respondents said "yes", while 
eleven percent said "no'\ 
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Four of those who responded negatively stated that the deciles were 
confusing. Three said that "statistical or computer things" always did 
confuse them* One person said he had trouble with the reversed or nega- 
tively phrased items, and one person said that the Analysis by Subscores 
section of the printout was not understandable* 

The next question was directed specifically at the interpretability 
of the deciles* Seventy-^five percent of the respondents said that the 
deciles were helpful in interpreting the results of the evaluations* 

The reasons given for saying the decile did not aid in the inter- 
pretation of the results were (1) they were too difficult to understand, 
(2) they might be abused by administrators, (3) the deciles are too precis 
given the unreliable nature of the data, and (4) it yas difficult to see 
how an item mean above 4*0 on a five point scale could result in a decile 
value of 2* 

The next question asked if the MIRROR evaluation forms were helpful* 
A large majority of the respondents (96^) said '*yes"* Only person 
elaborated upon his "no" response* He felt that the studen;^ were not 
sincere when they completed their forms* As an aside, 67 or 567# of the 
questionnaires had responses to this question, consequently this figure 
(567*) offers an approximation to the percentage of our users who Apt to 
participate in the MIRROR evaluation* 

The next question asked the respondent if the MIRROR forms were more 
informative than the computer printout* The response options were "yes", 
"no", and "both were equally informative* The favorable response given to 
MIRROR forms suggest that they should continue to be offered to consenting 
faculty members* 
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The final question was a global one which asked the faculty member 
if there was anything that could be done to improve the evaluation service. 
The recommendations offered in the order of their frequency were; 

(1) Develop campus norms based upon all instructors rather than 
voluntary participants, (Frequency 11) 

(2) Change certain items on the questionnaire. The items men- 
tioned most often were the negatively phrased items in the 
course evaluation section* (Frequency = 10) 

(3) Provide college-wide norms instead of university-wide norms 
(Frequency *2) 

(4) Provide a workshop for faculty on the use and interpretation 
of the IIQ* (Frequency ^ 2) 

On the basis of these results it appears that some action should be 
taken to (1) help faculty interpret the deciles, (2) provide college and/or 
departmental norms, (3) continue providing open-ended student feedback 
to faculty, (4) inform faculty of the availability of the optional item 
section on the IIQ, (5) provide some service such as workshops or feed- 
back sessions with instructors concerning the results of their evaluations 
and finally (6) encourage administrators to require full participation by 
faculty for the purpose of establishing truly representative norms for 
the IIQ, 
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1^ Name: 



2* Departaietit: 



3* Were you satisfied with the service you received from the Testing 
Center? 



Yes 



No 



If you were not satisfied with our service, please state why* 



4* Was the computer printout understandable? Yes I I No I I 

If the printout wasn't understandable, please state which part(s) 
was (were) not clear to you and why* 



5* Did the deciles help you to interpret your results? Yes | | No | ] 
If you didn't find the deciles helpful, please state why* 



6^ If you used the M irror evaluation forms (the open-ended or essay type 
form), did you find the comments of your students helpful? 



Yes 



No 
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Were the Mirror forms more informative than the computer printout? 





Yes 




No 





Both types of information 
were equally informative 



What could be done to improve the evaluation service? 
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A CROSSVALIDATION OF THE HQ DECILES 
FROM 1971 TO FALL 1972 

Mark VanTuinen and John Pohlmann 
, Southern Illinois University at Carbondale 

! ■ i 

The general purpose of this study was to assess the appropriateness 
o£ the Instjructional Improvement Questionnaire norms from 1971 for the 
, evaluations from Fall of 1972^ To that end the item means for the first 
forty items of the HQ were calculated for each of the participating ins** 
^ructors^ -This included about 900 instructors and 1454 sections for Wlnteri 
Springi SuTiimeri and Fall of 1971| and about 282 instructors and 446 sections 
for Fall of 1972. 

Upon rank ordering the item meansi and assuming that the 1971 item 
. means werfe fdistributed normally within each itemi nine decile values 

^,^£, (Istj 2nd ^ . , 9th^ for each item were obtained* This was done by converting 

each item mean to a? score using the normative item mean, and standard 
deviation of item means^ An item mean was placed in the 1st decile if its 
^ equivalent was less than *'1.28, which is the 1st decile value for the 2 
distribution. The item means for the Fall, 1972 data were also rankedi but 
in this case the nine decile values for each item were empirically determined^ 
That iS| an item mean was placed in the 1st decile if it fell in the lowest 
10% of item means^ There was no conversion to ? scores, and hence no 
assumption of normality with this procedure. Table 1 contains the decile 
cutoff points for each item for both the 1971 and Fall 1972 data* As can 
be seen in jcable 1, there was close correspondence between the two procedures 
for generating deciles* The mean absolute difference between decile values 
^ i for all forty items using the rwo systems was -06* The deviations were 
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most severe for items 29, 33, 18, and 21< 

The forty item means for each of the 1972 instructors were then com- 
pared to the 1971 and 1972 decile values. Thus, the mean score on item one 
for professor '*A'* would be located in the appropriate decile on both the 

1971 and 1972 item-one distributions. In this manner the percentage of 1972 
item means falling in the same decile in both the 1971 and 1972 distributions 
could be determined* The percentage of item means located in either identical 
or adjacent (e.g., located in the 4th decile on item one for the 1971 dis- 
tributions, and the 3rd or 5th decile on item one for the 1972 distribution) 
was also calculated. These percentages provided a measure of the accuracy 

of the 1971 norms for the 1972 data. The results of this analysis appears 
in Table 2. 

The results indicated that the item means for t^he two years are quite 
similarly distributed. The percentages for t:he item means falling in identical 
deciles in the two distributions ranged from 25. iT^ to 82.29%* On all but 
five of the items, the percentage of means located in either identical or 
adjacent deciles was 100* The two items for which this p&rcentage was 
lowest were itans 29(78.25%) and 33(78.03%). A comparison of the 1972 
means and decile values with the 1971 means and decile values for these 
two items indicated that the discrepancy was due to a downward shift in the 

1972 distribution (that is, the evaluations tended to be lower for these 
items in 1972), rather than a change in the shape of the distribution* 

In conclusion, this study indicated that the 1971 norms are generally 
appropriate for the 1972 evaluation results. It is, however, recommended that 
the norms for items 29 and 33 be updated to reflect the change In the rating 
distribution. These results also suggest that the assumption of normality. 
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made In the calculation of decile cut off values, is reasonable and tenable. 
This assumption wllL therefore continue to be used In the derivation of 
decile values. 
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TABLE 1. Decile values for the 1971 data appear as the 

first row for each Item. Decile values for the 
Fall 1972 data are In the second row of each 
Item. 



0.0 
0.0 



3.39 
3.39 



3.61 
3.72 



3.77 
3.93 



3.90 
4.04 



4.03 
4.16 



4.15 
4.26 



4.29 
4.36 



4.45 
4.50 



4.67 
4.68 



5.00 
5.00 



0.0 
0.0 



3.13 
3.13 



3.36 
3.41 



3.53 
3.61 



3.68 
3.76 



3.81 
3.89 



3.94 
4.04 



4.09 
4.16 



4.26 
4.27 



4.49 
4.43 



5.00 
5.00 



0.0 2.76 3.03 
0.0 2.75 3.00 



3.22 3.38 3.53 
3.21 3.42 3.55 



3.68 3.84 4.03 4.30 5.00 

3.69 3.91 4.13 4.36 5.00 



0.0 
0.0 



3.05 
3.08 



3.29 
3.33 



3.46 
3.55 



3.61 
3.73 



3.75 
3.87 



3.89 4.04 
4.00 4.10 



4.21 
4.26 



4.45 
4.43 



5.00 
5.00 



0.0 2.70 2.96 3.15 3.31 3.46 3.61 3.77 3.96 4.22 5.00 

0.0 2.64 2.93 3.19 3.37 3.53 3.64 3.81 3.93 4.15 5.00 

0.0 3.16 3.41 3.59 3.74 3.88 4.02 4.17 4.35 4.60 5.00 

0.0 3.12 3.47 3.67 3.79 3.96 4.10 4.23 4.33 4.52 5.00 

0.0 3.20 3.43 3.60 3.75 3.88 4.01 4.16 4.33 4.56 5.00 

0.0 3.13 3.45 3.67 3.83 3.97 4. 12 4.21 4.33 4.52 5.00 



0.0 
0.0 



3.75 
3.79 



3.94 
4.03 



4.08 
4.17 



4.20 
4.31 



4.31 
4.40 



4.42 
4.53 



4.54 
4.61 



4.68 
4.73 



4.87 
4.82 



5.00 
5.00 



0.0 3.23 3.45 3.61 3.74 3.87 3.99 4.13 4.29 4.51 5.00 
0.0 3.29 3.55 3.70 3.83 3.97 4.09 4.20 4.30 4.48 5.00 



10 



11 



0.0 
0.0 

0.0 
0.0 



2.99 3.23 
3.07 3.32 



2.89 
2.86 



3.16 
3.17 



3.40 3.55 3.69 3.83 3.98 4.15 4.39 
3.49 3.62 3.74 3.87 3.99 4.14 4.30 



3.35 
3.37 



3.52 
3.55 



3.67 
3.73 



3.82 
3.92 



3.99 
4.07 



4.18 
4.24 



4.45 
4.44 



5.00 
5.00 

5.00 
5.00 
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Average absolute differences between 1971 and 
1972 deciles for each Item. 

.09 
.06 
.04 
.06 
.05 
.06 
.06 
.08 
.07 
.06 
.04 



T 



'tablet.' beciie*^aiue8 ^or" tiie' li?5rdata'appear~*as" the 

first row for each Item, Decile values for the 
Fall 1972 data are in the second row of each item. 



12 0.0 3.92 4.10 4.23 4.34 4.^;4 4.54 4.65 4.78 4.96 5.00 
0.0 4.03 4.22 4.33 4.43 4.5C 4.59 4.66 4.75 4.35 5.00 

13 0.0 3.11 3.33 3.49 3.63 3.75 3.88 4.01 4.17 4.39 5.00 
0.0 3.10 3.38 3.56 3.69 3.83 3.97 4.12 4.25 4.44 5.00 

14 0.0 3.10 3.31 3.46 3.59 3.71 3.83 3.96 4.11 4.32 5.00 
0.0 3.02 3.31 3.50 3.64 3.78 3.91 4.04 4.20 4.38 5.00 

15 0.0 2.99 3.29 3.50 3.68 3.85 4.02 4.20 4.41 4.71 5.00 
0.0 3.04 3.33 3.54 3,76 3.98 4.14 4.32 4.46 4.64 5.00 

16 0.0 3.32 3.55 3.72 3.87 4.00 4.13 4.28 4.45 4.68 5.00 
0.0 3.32 3.53 3.76 3.97 4.07 4.20 4.33 4.48 4.69 5.00 

17 0.0 3.95 4.12 4.25 4.35 4.45 4.55 4.65 4.78 4.95 5.00 
0.0 3.88 4.17 4.32 4.42 4.53 4.63 4.69 4.79 4.87 5.00 

18 0.0 3.24 3.45 3.60 3.73 3.85 3.97 4.10 4.25 4.46 5,00 
0.0 3.42 3.58 3.75 3.88 3.96 "4,12 4.23 4.35 4.48 5.00 

19 0.0 3.13 3.37 3.54 3.69 3.83 3.97 4.12 4.29 4.53 5.00 
0.0 3.04 3.33 3.53 3.69 3.90 4.07 4.25 4.37 4.60 5.00 



•20 



0.0 3.10 3.37 3.56 3.72 3.87 4.02 4.18 4.37 4.64 5.00 
0.0 3.09 3.38 3.6* 3.88 4.00 4.20 4.30 4.46 4.58 5.00 



21 0.0 3.16 3.39 3.56 3.71 3.84 3.97 4.12 4.29 4.52 5.00 
0.0 3.19 3.51 3.70 3.84 4.00 4.11 4.25 4.39 4.55 5.00 

22 0.0 3.18 3.39 3.54 3.67 3.79 3.91 4.04 4.19 4.40 5.00 
0.0 3.18 3.47 3.65 3.76 3.90 4.02 4.12 4.27 4.44 5.00 



to 
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Xvefage absolute d^jf^rences between 1971 and 
1972 deciles for each item. 



.07 
.07 
.06 
.08 
.04 
.06 
.12 
.06 
.10 
.11 
.09 



TABLE 1. Decile vitlues for the 1971 data appear as the 

first row for each item. Decile values for the 
Fall 1972 data are In the second rov of each 
Item. 



23 0.0 3.12 3.35 3.52 3.66 3.79 3.92 4.06 4.23 4.46 5.00 
0.0 3.14 3.44 3.64 3.77 3.91 4.03 4.13 4.28 4.43 5.00 



24 
25 



0.0 
0.0 

0.0 

0.0 



2.69 2.93 3.10 3.25 3.39 3.53 3.68 3.85 4.09 5.00 
2.67 3.00 3.21 3.37 3.52 3.64 3.78 3.95 4.14 5.00 



3.09 
3.20 



3.30 
3.45 



3.45 
3.54 



3.58 
3.62 



3.70 
3.72 



3.82 
3,81 



3.95 
3.90 



4.10 
4.01, 



4.31 
4.17 



5.00 
5.00 



26 0.0 2.82 2.99 3.11 3.21 3.31 
0.0 2.99 3.11 3.22 3.37 3.37 



27 



0.0 
0.0 



2.83 
2.80 



3.10 
3.10 



3.29 
3.33 



3.45 
3.50 



3.60 
3.67 



3.40 3.51 3.63 3.80 5.00 
3.34 3.40 3.49 3.62 5.00 



3.75 
3.85 



3.91 
4.03 



4.10 
4.24 



4.37 
4.35 



5.00 
5.00 



28 0.0 3.10 3.32 3.48 3.61 3.73 3.85 3.98 4.14 4.36 5.00 
0.0 3.08 3.33 3.54 3.67 3.77 3.86 3.95 4.05 4.20 5.00 

29 0.0 2.46 2.76 2.98 3.16 3.33 -3.50 3.68 3.90 4.20 5.00 
0.0 2.14 2.42 2.67 2.90 3.05 3.22 3.44 3.67 3.95 5.00 

30 0.0 2.98 3.23 3.41 3.56 3.70 3.84 3.99 4.17 4.42 5.00 
0.0 3.03 3.33 3.52 3.66 3.78 3.88 3.96 4.09 4.24 5.00 

'31 0.0 3.26 3.49 3.66 3.80 3.93 4.06 4.20 4.37 4.60 5.00 

0.0 3.20 3.56 3.76 3.91 4.04 4.17 4.27 4.43 4.58 5.00 

32 0.0 2.33 2.60 2.80 2.96 3.12 3.27 3.44 3.64 3.91 5.00 

0.0 2.21 2.66 2.83 3.00 3.18 3.39 3.52 3.71 3.93 5.00 
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Average absolute differences between 1971 and 
1972 deciles for each Item. 

.08 
.09 
.08 
.07 
.06 

.05 w 

.27 

.09 

.08 

.07 



TABLE 1. Decile values for the 1971 data appear as the 

first row for each Item. Decile values for the 
Fall 1972 data are In the second row of each 
Item. 



33 



34 



35 



36 



37 



38 



39 



40 



0.0 2.50 2.72 2.88 3.01 3.14 3.26 3.40 3.56 3.78 5.00 

0.0 2.53 2.72 2.82 2.91 3.00 3.09 3.21 3.34 3.52 5.00 

0.0 2.57 2.82 3.00 3.15 3.29 3.43 3.58 3.76 4.01 5.00 

0.0 2.54 2.86 3.06 3.26 3.39 3.55 3.65 3.81 3.99 5,00 

0.0 3.10 3.31 3.46 3.58 3.70 3.82 3.94 4.09 4.30 5.00 

0.0 3.11 3.28 3.43 3.60 3.72 3.83 3.88 3.03 4.25 5.00 

0.0 3.07 3.27 3.42 3.54 3.66 3.77 3.90 4.05 4.25 5.00 

0.0 3.22 3.38 3.52 3.64 3.76 3.84 3.94 4.05 4.21 5.00 

0.0 3.61 3.80 3.94 4.05 4.16 A. 27 4.38 4.52 4.71 5.00 

0.0 3.63 3.83 4.02 4.09 4.21 4.32 4.43 4.50 4,69 5.00 

0.0 2.66 2.90 3.07 3.21 3.35 3.48 3.63 3.80 4.04 5.00 

0.0 2,47 2.84 3.05 3.25 3.39 3.50 3.65 3.82 3.99 5.00 

0.0 2.45 2.73 2.94 3.12 3.28 2.44 3.62 3.83 4.11 5.00 

0.0 2.39 2.66 2.93 3.18 3.41 3.54 3.68 3.85 4.04 5.00 

0.0 3.22 3.45 3.62 3.77 3.90 4.03 4.18 4.35 4.58 5.00 

0.0 3.22 3.47 3.73 3.92 4.00 4.13 4.27 4.42 4.58 5.00 



to 

00 



Average absolute differences between 1971 and 
1972 deciles for each Item. 
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NORMS FOR REQUIRED AND ELECTIVE 
COURSES BY COURSE LEVEL FOR THE 
HQ SU6SCALES 

John T » Pohlmann 
Mark Van Tuinen 
Testing Center 

The purpose of this study was Co develop norms for five subscalea 
from Che Instructional Improvement: Questionnaire (HQ) for required 
and elective couraes at each course level. The aubscalea of the HQ 
were derived from the resulta of an earlier factor analysis of items 
In parts I and II of the HQ* The five factors identified in that 
study were labled I, General Course Rating^ 2. Student Orientation^ 
3» Course difficulty^ 4. Grading and assignmenta^ and 5. Preaentalion 
of material. For the purpoaes of thia study items that loaded high on 
each factor were combined to form subacalea* Scorea on these 
subacales were derived by computing the average item acore for the 
items in the subscale. Conaequently five subacorea were generated 
for each student evaluation^ one for each subscale. In Table I the 
HQ items that were included in each subscale are listed. (The items 
in Parta I and II of the HQ appear in the Appendix) » 
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TABLE I 

THE HQ ITEMS THAT WERE INCLUDED IN 
THE FIVE SUBSCALES 



SUBSCALE HAME 


ITEM NUMBERS OF ITEIG IHCLUDED ' 


General Course Rating 


21,22,23,27,29,31,37,40 


Student Orientation 


10,16,18,19 


Course Difficulty 


25,36,38,39 , ^ . 


Assignments and Grading 


2,3,4,13,14 


Presentation of Material 


1,5,6,7,9,20 



The data used in this study were taken from ^ master research 
tape which contains the responses of 33,531 students to the IIQ> 
The data on the master tope apan the period from Winter 1971 to, 
and including Fall* For the purposes of this study, every other 
record on the tape was sampled^ Each sampled record was checked 
for mi'ising data, and if missing data were founds that record was 
deleted^ This process resulted in 11,639 cases suitable for 
statistical analysis* 
Statistical Analyses 

The initial phase of the statistical anslysis consisted of 
identifying subjects that fell into various categories according 
to course levels and required versus elective courses^ There were 
five possible course levels (100,200^ etc>)^ and at each course 
level students were identified who were either required to take * 
the course or took it as an elective* Students were identified 



4^ 
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as taking the course either as a requirement or as an elective on 
the basis of their responses to question 66 froa the HQ (Why did 
you take this course?)* If a student responded by saying that he , 
was taking the course as I* a personal choice^ or 2* an elective 
for a major or mlnor^ he vas categorized as taking the course as 
an elective* If the student responded to question 66 by stating 
his reason for taking the course vas to I* fulfill General Studies 
Requirements^ 2* fulfill requirements for major or mlnor^ or 
3* satisfy admission def Iclencles^ he was categorized at being 
required to take the course* Hence^ ten categories were Identified 
In a 2 X 5 (required versus elective) contingency table* Table 2 
presents the number of observations Included In each category* 

The second phase of the analysis consisted of computing the 
mean rating on each of the five subscores^ for each of the ten 
categories* 

TABUE 2 . 
FREQUENCY OF STUDENTS IN EACH 
OF THE CATEGORIES USED IN THE STUD! 

COURSE LEVEL I 





100 


200 


300 


400 


500 


Required 


2202 


1926 


3363 


763 


217 


. . . Elective 


398 


360 


1378 


887 


145 
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^ Results 

The following tables present the average scores for the five 
- subscales for each category, 

TABLE 3 
GENERAL COURSE RATXNG 
COURSE LEVEL 







100 


200 


300 


400 


500 




Required 


3.6 


3.7 


3.7 


3.8 


3. .9 


\ 


Elective 


4.0 


3.8 


3.9 


4.0 


4.3 








TABLE 4 












STUDENT ORIENTATION 












COURSE 


LEVEL 










100 


200 


300 


400 


500 




Required 


3.8 


3.8 


3.8 


3.9 


4.1 




Elective 


3,9 


3.7 


3.9 


4.0 


4.2 








TABLE 5 












COURSE DIFFICULTY 
COURSE LEVEL 










100 


200 


300 


400 


500 


.V 


Required 


3.4 


3.4 


3.5 


3.5 


3.4 


rv 


Elective 


3.5 


3.6 


3.6 


3.7 


3.7 
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TABLE 6 
ASSIGMHEMTS AND GRADING 
COURSE LEVEL 





100 


200 


300 


400 


500 


Required 


3.7 


3.7 


3.6 


3.7 


3.7 


Elective 


4.0 


3.8 


3.8 


3.9 


4.2 



— 

TABLE 7 
PRESENTATION OF MATERUL 
COURSE LEVEL 





100 


200 


300 


400 


500 


Required 


3.8 


3.8 


3.8 


3.8 


4.0 


Elective 


4.0 


3.8 


3.8 


3.9 


4.2 
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f ART I: INSTRUCTOR EVALUATION: ITEMS 1 THROUGH 20 



g 
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DIRECTIONS: Tha following -twenty phrvsec r«Tale to coll«0e*l«v«l teaching. 
Ev«lum how your irv»lructCK did in Mch of th«s« Mpect* of teaching by s«l«ctir>9 
tht ort# rnponw option W throuDt> £ below) that com** ciQS9%X to your 
judgment. 
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RESPONSE OPTIONS: OMIT ITEMS THAT DO NOT APPLY 

A. Exception^ or outstanding performince 

6. Vetygood performance 

C. Good performance, all that I would normally expect in college-level 
teaching 

0. Wesk performance, instructor shouFd be aware of some opportunity for 
fnrprovement 

E. fmpfo^ement definiMy needed 

1. Prepared for class 

2. Made clear assignments 

3. Set clear standards for grading 

4. Graded fairly 

5. Knew if students understood him 

6. Spoke understandably 

7. Answered impromptu questions satisfactorily 
B. Showed an interest in the course 

9. Gave several examples to explain complex ideas 

10. Accepted criticism and suggestions 

1V Increased your appreciation for the subject 

12. Was dependable in holding class as scheduled 

13. Speciffed objectives of the course 

14. Achieved the specified objectives of the course 

15. Promptly returned homework and tests 

16. Showed an interest in students 

17. Made assignments that heJped you understand the course 

18. Was available outside of class 

19. Encouraged participation of students 

20. - In general, taugiht the class effectively 



MRT II : SVALUATION OF COURSE: iTEfW 21 THROUGH 40 



DIRECTIONS: In the fotlowirtg twenty ttat«fn«nt*. tndicatt your f««tir>9 about 
thtt cour«« by selecting ott* of tht r«9pon*t optton* iA through f below). 



RESPONSE OPTIONS: OMIT ITEMS THAT DO NOT APPLY 

A. I STRONGL Y AGREE with tfiis statement, 

8. I AGREE with this statement, 

C. I can NEITHER agree nor disagree with this statement. 

0, I DISAGREE with this statement, 

E, I STRONGL Y DISAGREE with this statement. 



Soma quastions below are worded in the opposite dir«ction-/?ea^ f ad^ Iteftt 
Cerefully* 

Start With Item Number 21 on the Answer Sheet 

21. Thiscourse wasa good learning experience for me. 

22. The content of this course was good. 

23. The course was well organized. ^ 

CO 

24. I had troubte paying attention in class, 

25. There should be additional prerequisites for tiiis course. 

26. There should be fewer prerequisites for this course. 

27. This course was very interesting. 

28. The amount of required work was appropriate. 

29. This was one of the better courses 1 have taken. 

30. The tesU covered the course material well. 

31 . This course was a waste of time. 

32. The textbook was good. 

33. This course could have used audio-visuals more effectively. 

34. This course should be taught fn some other way. 

35. I covered much of this material in other courses. 

36. The course material was too difficult. 

37. This course should continue to be offered. 

38. The reading assignments were hard to understand. 

39. I was often confused. 

40. Generally, the course was good. 
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FACTOR ANALYSES OF PARTS I AND II 
OF THE HQ 

John T» PoMraann 
Teotlng Center 

Data Source 

The data for this study came from a master research file which 
contains the records of all student evaluations using the HQ from 
the Winter Quarter 1971 to the Fall Quarter 1971* A total of 
33|53l records are contained on the master file* For the purposes 
of this study every tenth record was read and checked for missing 
data* If a record with missing data was encountered^ the record 
was deleted from the study* This process resulted In 2»5?5 and 
2^447 cases for the factor analysis of Parts I and II of the IIQ» 
respectively* 

Factor Analyses 

A correlation matrix for each of Parts I and II was calculated* 
Squared multiple correlations (SMC) for each Item^ using the other 
Items as predictors^ was placed In the main diagonal of the 
correlation matrix* The resulting matrix of Item Intercorrelatlons 
and SMC^s was then factor analyzed using principle axis analysis* 
Tables I and 2 present the results of the principle axis analyses 
for Parts I and II respectively for all factors with Eigenvalues 
greater than 1*0* 
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TABLE 1 

RESULTS OF THE PRIMCIPLE AXIS ANALYSIS OF PART I OF THE HQ 





ITEH 


I 


FACTOR 
II 


III 


Comimmality 


1, 


Prepared for class 


68* 


15 


-01 


49 


2, 


Made clear assignments 


66 


40 


03 


60 


3, 


Set clear standards for 












grading 


63 


50 


-03 


65 


4, 


Graded fairly 


70 


23 


-07 


55 


5, 


Knew if students understood 












him 


76 


-01 


30 


67 


6, 


Spoke understandably 


71 


-10 


29 


59 


7, 


Answered impromptu 








* 




questions satisfactorily 


75 


-12 


26 


64 


8, 


Showed an interest in the 












course 


70 


-24 


-09 


54 


9, 


Gave several examples to 












explain complex ideas * 


72 


00 


18 


59 


10. 


Accepted criticism and 












suggestions 


74 


-13 


08 


56 


11, 


Increased your appreciation 












for the subject 


76 


-11 


19 


62 


12. 


Was dependable in holding 












class as scheduled 


50 


-09 


-52 


52 


13, 


Specified objectives of 












the course 


72 


23 


-04 


57 


t 


AchlfiVfid thfi fiDficf.fl fid 

-nblt V kite 0|^Cb X 












objectives of the course 


78 


19 


-02 


64 


15, 


Promptly returned 












homework and tests 


54 


16 


-54 


61 


16, 


Showed an interest in 












students 


76 


-30 


-13 


69 


17, 


Made assignments that 












helped you understand the 












course 


74 


05 


00 


54 


18, 


Was available outside of 












class 


63 


-29 


-29 


57 


19, 


Encouraged participation 












of students 


69 


-41 


-07 


65 


20, 


In general^ taught the 












class effectively 


85 


-04 


12 


73 




Eigenvalue 


9,92 


1,06 


1,01 






Percent of Total Variance 


49,6 


5,3 


5,0 





* Decimal points have been removed from the factor loadings and 
communal it ies. 
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TABLE 2 • 

RESULTS OF THE PRINCIPLE AXIS ANALYSIS OF PART II OF THE HQ 





ITEM 


I 


II 


FACTOR 
III 


IV 


ConraunallCy 


21. 


This course was a good 














learning experience for toe 


. 82* 


22 


09 


12 


75 


22. 


The content of this course 














was good. 


80 


23 


11 


02 


70 


23. 


The course was well 














organized. 


00 


1 Q 

lo 




-17 


50 


24. 


I had trouble paying 














attention In class. 


-56 


08 


26 


-21 


43 


25, 


There should be additional 












- 


prerequisites for this 














course. 


-19 


60 


03 


•^40 


56 


.26. 


There should be fewer 














prerequisites for this 














course. 


-06 


-07 


63 




61 


27. 


This course was very 














Interesting. 


82 


22 


08 


15 * 




28. 


The amount of required work 












was appropriate. 


55 


-07 


12 


•*17 


35 


29. 


This Was one of the better 














courses I have taken. 


84 


18 


10 


07 




30. 


The tests covered the 














course material well. 


53 


10 


00 


-27 


'37 ^ _ 


31. 


This course was a waste of 














time. 


-78 


-03 


12 


-15 


65 


32. 


The textbook was good. 


37 


06 


29 


52 


49 


^33 > 


This course could have 












- 


used audlo-vlsuals more 














effectively. 


-25 


18 


55 


09 


41 


-34. 


This course should be 














taught In some other way> 


-73 


07 


25 


-02 


60 




I covered much of this 














material In other 














courses > 


28 


-09 


57 


-07 


41 


^ 36. 


The course material was 














too difficult^ 


-46 


66 


-02 


04 


65 


37. 


This course should continue 














to be offered. 


70 


10 


06 


06 


51 


'38. 


The reading assignments 














were hard to understand. 


-41 


60 


-14 


33 


66 


-39. 


I was often confused. 


-53 


58 


-13 


12 


65 


-40. 


Generally^ the course 














was good. 


86 


19 


07 


09 


78 ' 

] 




Eigenvalue 


7.38 


1.82 


1.34 


1.03 






Percent of Total Variance 36.9 


9.1 


6.8 


5.2 





* Decimal points have been removed from the factor loadings and the 
connmnalltlas . 
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Inspection of the principle axis factor loadings Indicates 
that both Parts I and II have a rather strong general factor* The 
high latent roots associated with the first factor In each solution 
attest to this fact* 

The Initial factors were rotated using the Varlmax criterion* 
The rotated factor structures for Parts I and II are presented In 
Tables 3 and 4» respectively* 

Interpretation of Rotated Factors 

Part I ; Factor I appeared to be measuring the clarity of 
coniBunl cation that existed In the course* Items that loaded high on 
this factor (5» 6» and 7) Indicated that high scores on this factor 
would be obtained by Instructors who spoke cl^arly^ answered student 
questions satisfactorily and knew when their students understood the 
Instructor* 

Factor II suggested Itself as a class management factor* Items 
that loaded high on this factor (2» 3» 13» and 14) Indicated that an 
Instructor who scored high on this factor would be one who made 
clear assignments^ set clear grading standards^ specified and 
achieved the objectives of the course* 

The Items that loaded highest on Factor III (12» 15» and 18) 
Indicated that Factor III was a dependability factor* An Instructor 
who scored high on this factor would be characterized as one who was 
dependable In holding claf,s as scheduled^ prompt In returning 
homework and testf^ and was available outside of class time* 

Part II ; Factor I appeared to be measuring the attitude of the 
student toward the course In general* Courses receiving high ratings 
on this factor were perceived as having been a good learning experience^ 
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having good content^ and having been one p£ the better courses to 
which the students have been exposed^ 

Factor XX had high loadings on items that related to the diffi* 
culty of the course (36^ 38> and 39), Courses with high scores on 
this factor were characterized as confusing^ having difficult 
inaterial> and in need of additional prerequisites. 

Factor XXX had high loadings on items (26 and 35) that indicated 
the course overlapped with other courses the students had taken. 

Factor XV remained as a specific factor evaluating the 
textbook. 
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TAELF. 3 

ROTATED FACTOR MATRIX FOR PART I OP THE HQ 



FACTOR 







T 
1 


TT 

xi 


TTT 
ill 


1* 






t;t 






n Aoc c ie«r «ssxgninenc s 




/v 




J* 












grading 


IV 


it 


15 


A 

H *■ 


ursae a r ax r ijt 






*v 




KncfW 11 scuoGnca unaer** 










n4tn 

scooQ nun 


/v 


Hi; 


y/ 




t> poKe un oe r s c anoaD ly 


7A 

/u 




uo 


7. 


Answered Impromptu 










questions s Atlsfactorlly 


11 




u 


o* 


Showed M Interest In the 










course 


^0 






9* 


Gave several examples to 










explain complex Ideas 




41 


15 


10* 


Accepted criticism and 










suggestions 


62 


31 


29 




Itxreased your appreciation 










for the subject 


62 


34 


20 


12* 


Was dependable in holding 










class as scheduled 


13 


21 


68 


13. 


Speclried objectives or 










the course 


37 


60 


27 


14. 


Achieved the specified 
ot>Jectlves of the 










course 


45 


60 


29 


15. 


Promptly returned home* 










work and tests 


02 


44 


64 


16. 


Showed an Interest In 










students 


62 


18 


53 


17. 


Made assignments that 
helped you understand 










the course 


49 


46 


30 


18. 


Was available outside of 










class 


43 


12 


61 


19. 


Encouraged participation 










of students 


65 


05 


48 


20. 


In general^ taught the 










class effectively 


68 


45 


27 



* Decimal points have been omitted from the factor loadings. 
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TABLE 4 

ROTATED FACTOR MATRIX FOR PART II OF THE HQ 



FACTOR 

ITEM I II III IV 



21 t This course was 4 good 

learning experience for me 

22, The content of this 

course was good, 

23, The course was well 

organized, ^ 

24, I had trouble paying 

attention In class, 

25, There should be additional 

prerequisites for this 
course, 

26, There should be fever 

prerequisites for this 
course, 

27, This course was very 

Interesting, 
28* The amount of required 

work was appropriate* 
29, This was one of the 

better courses I have 

taken, 

30* The tests covered the 
course material well, 

31f This course was a waste 
of time* 

32, The textbook was good, 

33, This course could have 

used audlo-vlsualfi more 
effectively, 

34, This course should be 

. taught In some other way, 

35, I Covered much of this 

material In other courses* 
36* The course material was 

too difficult* 
37* This course should con*- 

tlnue to be offered, 
38, The reading assignments 

were hard to understand* 
39* I was often confused, 
AO* Generally* the course 

was good* 



86* 


-09 


-03 


08 


81 


-08 


t02 


18 


62 


-07 


-12 


31 










-50 


24 


2$ 


18 


-06 


62 


-02 


41 


09 


-11 


72 


-25 


86 


-09 


-02 


05. 


44 


-26 


-03 


29 


84 


-13 


-03 


13 


45 


-09 


-16 


36 


-74 


23 


21 


04 


26 


-10 


07 


64 


-09 


20 


60 


04 


-62 


29 


36 


-03 


-25 


-05 


57 


■ 17 


-20 


78 


07 


02 


69 


-16 


-05 


09 


-10 


72 


04 


-36 


-27 


74 


02 


-19 


87 


. -13 


■ -06 


11 



* Decimal points have been omitted from factor loadings, 
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EVALUATION OF TKE 
INSTRUCTIONAL IMPROVEMENT QUESTIONNAIRE (HQ) 
BY FACULTY - FALL 1971 



Introduction 



During Winter Quarter of 1972^ <]uestlonnalres were sent to all 
those faculty who used the Instructional Improvement Questionnaire 
(HQ) during Fall Quarter of 1971, The quenf lonpalre was distributed 
with the results of the HQ analysis for each Unstructor* The 
purpose of the questionnaire was to determine the faculty^s response 
to the IIQ» as well as to seek suggestions as to the Improvement 
of the IIQt Although a similar study was made previously^ It was 
decided that a more comprehensive analysis ^as necessary* Of the 
427 questionnaires distributed, 102 were returned. 

Results 



The first two questions were concerned with a description of the course 
and Instructor, Of those faculty responding to the questionnaire 
(102 total responses): 

7% of the respondents were teaching assistants, 
16% of the respondents were Instructors, 
37% of the respondents were assistant professors, 
24% of the respondents were associate professors, 
16% of the respondents were professors. 

In addition, those responding Indicated the nature of the classes as 
follows: 

28% lecture, 
6% laboraf.ory, 

13% combination lecture and laboratory, 

33% combination lecture and discussion section, 

18% other. 



To the question of 


"helpfulness" of the HQ: 


25% 


indicated 


it 


to 


be 


"very helpful." 




indicated 


it 


to 


be 


"moderately helpful. 


271 


indicated 


it 


to 


be 


"somewhat helpful." 
"not helpful." 




indicated 


it 


to 


be 


ly. 


indicated 


it 


to 


be 


a "waste of time." 



To the question of the ease of Interpretation of the results of the HQ: 
67% Indicated the results were "easily understood,*' 
19% Indicated the results were "somewhat difficult to understand," 
13% indicated the results were ^'confusing in some parts," 
1% indicated the results were "difficult to understand," 
0% Indicated the results were "incomprehensible," 
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Of those responcMng: 

91% Indicated that they Intend to. submit a copy of the HQ 
results to their department chairman* 
87* Indicated that they did not Intend to submit a copy of 

their results to their department chairman* 
3% omitted the Item* 

To the ques*-ijn "Do you Intend to use the evaluation again?'*: 
927* answered **y-s*** 
7% answered "no*" 
3% omitted the item* 

Of those snswerlng '*no*^ to this ltein> most were concerned 
with the "appropriateness*' of the HQ in terms of their own specific 
classroom situation* Uovever> they indicated that if additional 
questions or a new form was designed to meet their particular 
situation^ they would then use it* 

Vfhen asked with whom they Intend to compare their results; 

407* of the respondents Intended to compare their results with 

departmental norms that were sent to the department chairman* 
2^7* of the respondents Intended to compare their results with 

the course level norms that were sent to the department 

chairman* 

177* of the respondents Intended to compare their results wl^h 
the results of their colleagues for the same or similar 
course* 

467* of the respondents Intended to compare their results with 

their own previous evaluations for the same course* 
18% of the respondents did not Intend to make comparisons* 

When asked to respond to the statement results of the HQ 

indicated an accurate perception of my teaching in the course**'; 

13% responded **Yes, definitely*" 

707* responded "Yes, generally*** 

15% responded **Yes, to a limited extent*" 
07* responded '*No relationship**' 



When asked if they would like special rating forms developed: 

7% would like a special form developedfor laboratory sections* 

97* would like a special form developed for discussion sections* 

8% would like a special form developed for seminars* 

25% would like a special form developed for their specific 
teaching area* 

127* would like a special form developed for other circumstances* 



The respondents were asked to rank the parts of the IIQ> from 1 to 5> 
according to their utility JLn course eva3*jatlon* The following are 
the resulting mean scores (x) for each of the five parts: 

X 

1*6 Part *'Instrucl;or Evaluation" 

2*0 Part II: "Evaluation of Course" 

2*3 Part III; '^Strengths and Weaknesses'* 

3*3 Part IV: "Research Data*' 

4*2 Part V: *'Optlonal Items" 
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SUMMARY OF RESEARCH ON THE 
RELATIONSHIP BETWEEN STUDENT CHARACTERISTICS 
AND STUDENT EVALUATIONS OF INSTRUCTION 
AT SOUTHERN ILLINOIS UNIVERSITY, CARBONDALE 

Introduction 

One problem that typically arises in the interpretation of 
rating form results is the possibility that they might be 
influenced by characteristics and biases of the rater rather 
than those of the object or person rated. In the case of student 
evaluations of instruction, such factors might include student 
characteristics such as sex, year in school, level of academic 
achievement, etc. Since these characteristics are, generally 
speaking, beyond the control of the instructor, it vould be 
unfair to the instructor if he received a ''bad" evaluation 
because of a unique combination of such characteristics in his 
students. It would bo correspondingly unfair if he received a 
*'good" evaluation due to a unique combination of characteristics 
in his students that predisposed them to offer an overly 
positive evaluation. Ideally, instructors and administrators 
coitld interpret student evaluations more meaningfully if they 
could identify and take Into consideration those factors that 
tend to bias student ratings of instructors and courses. It was, 
therefore, the purpose of this study to determine the nature and 
degree of relationship th&t exists between certain student 
characterliitics; and student ratings of instruction. 
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Method 

Student Evaluations : The student evaluations that were used 
as criteria in this study were obtained from the Instructional 
Improvement Questionnaire (HQ), The HQ is a 72"-item questionnaire 
designed to provide evaluative feedback to instructors who elect 
to use it in the evaluation of their courses, There are five 
parts to the HQ. They are: 

I* an instructor evaluation section (20 items) , 

2» a course evaluation section (20 items) ^ 

3» a strengths and weaknesses section composed of 20 
forced choice items for identifying strong and weak 
points of the instructor and course, 

A* a research data section composed of 12 items that 
solicit descriptive data from the student, and 

5- an optional instructor supplied item section (28 
items maximum) that may be used by the instructor 
to evaluate special aspects of his course* 

Previous research on Parts I and II of the HQ suggested that there 

are throe separate aspects of course evaluation measured by these 

forty items. A factor analysis of the items in Parts I and II 

revealed three factors* They were identified as: 

1. an instructor evaluation factor, 

2* a course evaluation factor, and 

3. a course difficulty factor. 
For the purpose of this study, the items that loaded highest on 
each of theso throe factors were considered as separate scales, 
The item scores for each student respondent on each of these scales 
were £;ummed to obtain a total score for each scale. This total score 
was thon rescaled by dividing it by the number of items in the 

r 

scale. Thus, the rescaled score represents an average item score 
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for each scale < 

The means and standard deviations of the raw and rescaled 
scores, together with scale intercorrelations and reliability 
estimates appear in Table I. In the analyses that follow^ the 
three rescaled scorct; for each student respondent (Instructor 
Evaluation^ Course Evaluation^ and Course Difficulty)^ were used 
as criterion variables. Four of the items in Parts I and II 
did not relate to any of the three evaluation factors, consequently 
thev were not used In this study. 

Student Characteristics Data : The data used to determine 
the student characteristics which were examined in this study 
were obtained from Part IV of the HQ (Student description 
section). A copy of this section of the HQ appears in Appendix I. 

Student Sample : The student sample for this study consists 
of 811 students \^o completed the HQ in the Winter Quarter, 1971, 
They were selected by sampling approximately every tenth student 
who participated in the course evaluation program throughout 
the University during that quarter. 
Results 

In order to determine if any relationship existed between 
the Ktudent characteristics assessed by Part IV of the HQ and 
the three evaluation criteria, a One^'Way Analysis of Variance 
wa«; computed for each item on each of the evaluation criteria. 
The groiip«; in the Analyt;is of Variance were determined by which 
rei;pont;e option wa^; (selected by the student in each item in 
Part IV of the HQ. Tables 2 through 13 present the results 
of these analyses. Bach of the^e Tables contains the mean rating 
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TABLE 1 



MeanSi Standard Deviations^ Reliability Estimates and Intercorrelations 
of the Three Scales of Evaluation of Instruction 



Scale 



Number 
of 
Items 



Raw 
Score 
Mean 



Raw Score 
Standard 
Deviation 



Rescaled* 
Mean 



Re scaled* 
Standard 
Deviat ion 



Intercorrelations 
and Reliability 
Estimates 
I 2 3 



I. Instructor 
Evaluation 



20 



75.40 



14.60 



3.77 



.73 



.94** 



2. Course 
Evaluation 

3. Course 
Difficulty 



12 



40.80 



10.56 



7.32 



3.28 



3.40 



2.64 



.61 



*82 



.69 



.18 



.79 



,32 



*7l 



*Rescaled scores were obtained by dividing each subject's raw score by the number of items In the scale. 
**Rellabllity estimates appear In the main diagonal and are Cronbach's Alpha coefficients. 
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scale value of each of the evaluation scales (Instructor, 
Course^ and Course Difficulty) for each of the response options, 
the proportion of variance accounted for in each of the criteria, 
and the Univariate F-test for the Analysis of Variance, It 
should be mentioned, at this point, that the tests of signifi^ 
cance reported In these analyses can be somewhat misleading 
due to the large number of degrees of freedom in the error term 
for each test* Therefore, the reader is encouraged to pay close 
attention to the proportion of criterion variance accounted 
for by these analyses, since it is unaffected by sample size. 

Table 2 depicts the relationship between student sex and 
ratings of instruction. Only the Course Difficulty Scale was 
found to be significantly related to the sex of the student. 
Female students tended to rate courses as being more difficult 
than their male classmates. Even though a significant relation- 
ship existed between student sex and rating of course difficulty, 
only one percent oC the variance in the Course Difficulty 
criterion was accounted for by knowledge of student sex* 

Table 3 shows the relationship between student^s year in 
school and ratings of instruction^ Again the only criterion 
which i:^ appreciably related to student's year in school is the 
Cour:;e Difficulty scale. The data suggests a general negative 
trend between year in school and Course Difficulty, freshmen 
rating the courses a.'^ most difficult and graduate students rating 
courtiet; as least difficult. 

Table A presents the relationship between the level of the 
cour:;e (100, 200, etc.) and ratings of instruction. Course 
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TABLE 2 

The Relationship Between Sex of Student and Ratings of Instruction 



Rating Scale 



Sex of Student 



Male 
n=585 



Proportion of 
Variance in 
Female Rating Accounted 
n=212 for by Sex 



Univariate 
F test 



Instructor Evaluation 



3.79 



3.76 



.001 



2.45 



Course Evaluation 



3.52 



3.47 



.002 



3.57 



Course Difficulty 
High Score = Difficult 



2.61 



2.70 



.010 



8.1** 



** P -C .01 



TABLE 3 
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The Relationship Between Student's Year In School and 
Student Ratings of Instruction 



Rating Scale 



Student's Year in School 

Freshmen Sophomores Juniors Seniors Graduates 
n = il4 n = 161 n = 279 n = 225 n = 32 



Proportion of 
Variance in 
Ratings Accounted 
for by Year in School 



Univariate 
F test 



Instructor Evaluation 



3.73 



3.82 



3.76 3.77 



3.85 



.002 



.43 



Course Evaluation 



3.59 



3.60 



3.58 3.61 



3.76 



.003 



.74 



Course Difficulty 
High Score = Difficult 



3.89 



2.67 



2.64 2.55 



2,33 



.023 



4.77*** 



*** P < .001 



TABLE 4 
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The Relationship Between Course Level 
and Student Ratings of Instruction 



Rating Scale 



100 
n=193 



200 
n=180 



Course Level 

300 400 
n=329 n=92 



500 
n=l3 



Proportion of 
Variance in 
Rating Accotinted 
for by Course 
Level 



Univariate 
F test 



Instructor Evaluat Ion 3 . 76 3 . 69 3 . 80 3.81 3. 74 



.006 



1.14 



Course Evaluation 



3.55 3.52 3.65 3.65 



3.74 



.013 



2.65* 



Course Difficulty 
High Score = 
Difficult 



2.91 2.65 2.50 2.53 2.74 



.041 



8.61*** 



* p < .05 
*** p <. .001 
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Evaluations tend to improve as the course level increases and 
ratings of Course Difficulty tend to decrease as course level 
Increase*;. The higher rating of Course Difficulty for gra()uate 
students is probably due to sampling variability since only 
thirteen students were enrolled in graduate level courses. An 
independent t test was calculated to test this hypothesis^ 
and was found to bo non-significant (t=l.l2, df^\2) * 

In the analyses where GFA^ Expected Grade and Deserved 
Grade were considered, only undergraduates were used in the 
analyses, since graduate students did not vary on these dimensions* 
All graduate students had GFA's above 4.0, expected A's, and 
felt that they deserved a's. The relationship between reported 
GPA and student evaluations of instruction Is presented in 
Table 5. No relationships were found between student GFA ^i^d 
any of the evaluation criteria. 

In Table 6, the relationship between the grade expected 
by the student, and ratings of instruction is presented. 
Expected grade was found to be most strongly relate() to the 
rating of Course Difficulty, and accounted for twelve percent 
of the variation In that rating. As would be expected, the 
perceived difficulty of the course increases as the expected 
>;rade decreases. Ratings of the instructor and course tend to 
ineroaso as the expected grade increases. Although impressive, 
clu.^ result does not support a causal hypothesis about the effects 
oi expected grade on evaluations. All that can be said, at this 
point, i& that a relationship exists between these two Indices. 

Table 7 presents the relationship between the grade the 
student feels he deserves and ratings of instruction. Deserved 
Q grade was most strongly related to the rating of course difficulty. 
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TABLE 5 



The Relationship Between Student's Reported 
Grade Point Average and Ratings of Instruction 
(Undergraduate Students Only) 



Rating Scale 



4.CK- 
n=l47 



Reported GPA 



3.S to 3.9 
n=244 



3.0 to 3.4 

n=323 



2.0 to 2.9 
n=62 



2.0- 
n=3 



Percent of 
Variance In 
Rating Accounted 
for by GPA 



Univariate 
P-test 



Instructor Evaluation 3.76 



Course Evaluation 



Course Olfflculty 
Histh Score = 
Difficult 



3.36 



2.63 



3.74 
3.34 
2.63 



3.79 



3.43 



2.68 



3.74 



3.36 



2.68 



3.74 



3.36 



2. SO 



.001 



.003 



.001 



.29 
.88 
.19 



CD 
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TABLE 6 



The Relationship Between the Grade Expected 
by the Student and Ratings of Instruction 
(U lid er graduate Students Only) 



Rating Scale 



Grade Expected 
A B C D E 

n=167 n=354 n=207 n=43 n=8 



Proportion of 
Variance In 
Rating Accounted 
for by Expected 
Grade 



Univariate 
F-test 



Instructor Evaluation 



Course Evaluation 



Course Difficulty 



3.96 



3.55 



2.46 



3.83 



3.46 



2.47 



3.59 



3.23 



2.95 



3.41 



3.01 



3.41 



3.41 



2.96 



3.21 



.048 



.066 



.123 



9.70*** 
13.62*** 
27.11*** 



*** P < .001 
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TABLE 7 



The Relationship Between the Student's Reported 
Deserved Grade and Ratings of Instruction 



Rating scale 



A 

n=230 



Grade Deserved 

B C 
n-357 n«15& 



D 

n=30 



E 

n=6 



Proportion of 
Variance In 
Rating Accounted 
for by Deserved 
Grade 



Univariate 
F-test 



Instructor Evaluation 3.85 



3.79 



3.63 



3.58 



3.58 



.015 



2.85* 



*^ Course Evalui -ion 



3.45 



3.45 



3.20 



3.12 



2.86 



.040 



8.06*** 



Course Difficulty 
High Score = 
Difficult 



2.43 



2.59 



3.00 



3.12 



2.83 



.074 



15,36*** 



* p < .05 
*** p < .001 



to 
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The Difficulty rating tended to increase as deserved grade 
decreased* On the other hand» a positive relationship was 
noted between deserved grade and ratings of the instructor 
and the course* These results were very similar to those 
observed for the "Expected Grade" variable because of the high 
correlation between "Expected Grade*' and "Deserved Grade" 
(r = *71)* 

In Table 8» the relationship between the responses of 
students to question 66 from the HQ and ratings of instruction 
is presented* The purpose of question 66 was to determine why 
the student was enrolled In the course* Courses that were 
personal choice electives were given the highest rating and 
were rated as least difficult* Courses taken to satisfy 
admissions requirements received the lowest ratings* Courses 
required for major or minor concentrations » and for General 
Studies were rated as most difficult* 

Table 9 depicts the relationship between reported class 
attendance and ratings of instruction* There was a positive 
relationship between attendance and course evaluation* A 
negative relationship was noted between course difficulty and 
attendance* The rating of the instructor was found to be 
independent of reported class attendance* 

In Table 10» the relationship between the student's 
transfer status and ratings of instruction is presented* Only 
the Course Difficulty scale was found to be related to student 
transfer status* Transfer students who spent more than two 
years at another institution rated their courses as least 
difficult* Students who had done work at the Edward^^ville Canpus 

Er|c 72 



TA&t£ 8 



The Relationship Between the Responses of Students 
to Question 66 from the HQ to Ratings of Instruction 



Question 66 


Response Options 


Instructor ' 
Evaluation 


Rating Scale 

Course 
Evaluation 


Course 

Difficulty 

High Score « Difficult 


Why did you 
take this 
course? 


A* 


Fulfill General Studies 
Requirements 
n - 208 


3.73 


— T 


3.30 


2.69 




B* 


Required for Major or Minor 
n = 423 


J* / / 




3 .39 






C* 


Elective for Major or Minor 
n = 73 


2*79 




3.47 


2.53 




D* 


Satisfy Admission Deficiencies 
n = 9 


3*43 




3.06 


2.39 






Personal Choice Elective 
n - 98 


3*89 




3.53 


2.31 




Proportion of Variance in Etating 
Accounted for by Choice Categories 


*007 




.017 


.028 




Univariate F*test 


1*37 




3.42** 


5.61*** 



** p < .01 
*** p < .001 



TABLE 9 



The Relationship Between Class Attendance 
on Ratings of I">struction 



Frequency of Attendance 



Rating Scale 



50X- 
n=i4 



50 to 80% 
n=60 



80 to 90% 
n=93 



Proportion of 
Variance in 

90 to 95% 957tff Ratings Univariate 

n=l74 n=470 Accounted for F-Test 
by Attendance 



Instructor Evaluation 3.95 



Course Evaluation 



Course Difficulty 
High Score =■ 

Difficult 



3.65 



3-19 



3.61 
3,13 
2.66 



3.69 



3.32 



2.87 



3.73 
3.39 
2.65 



3.82 



3.42 



2.57 



.008 



.022 



,021 



1.61 



4.43** 



4.34** 



P < .01 
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The Relationship Between Student's Transfer 
Status and Ratings of Instruction 



Instructor Course Course 
Evaluation Evaluation Difficulty 



A. All college work at S.I.U. Carbondale 
n = 415 



3,78 



3,37 



2,67 



B, All college work at S.I,U, (all campuses) 
n = 51 



3,66 



3,43 



2,86 



Transfer C, Less than one academic year completed at 

another college (s) 
Status n = 72 



3,71 



3,28 



2,84 



D, From one to two academic years at another 
college (s) 
n = 212 



3,78 



3.43 



2,50 



E, More than two academic years at another 
college (s) 
n = 57 



3,79 



3,39 



2,42 



Proportion of Variance in Ratings Accounted for 
by Transfer Status 



,002 



,004 



,025 



Univariate F-test 



,44 



,79 



5,01*** 



*** 



p < ,001 
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and students vho had done less than one year of work at another 
institution rated their courses as most difficult* 

Table II presents the relationship between the percent of 
assigned readings completed by the student and ratings of 
instruction* This variable had its highest relationship with the 
Course Difficulty scale* Students who completed 90 percent or 
more of the readings rated their courses as least difficult* 
There was also a tendency for students who completed 90 percent 
or more of the assigned readings to rate their instructors 
and courses more favorably* 

Table 12 presents the relationship between the number of 
hours outside of class spent studying and ratings of instruction* 
As the number of hours spent studying increased^ the ratings of 
instructor and course tended to be more favorable* Students 
who spent less than one or more than eight hours studying 
tended to rate their courses as being most difficult* Initially 
thefie results suggested that a second degree or U^'shaped relation* 
ship exifited between hours studied and rating of course difficulty, 
but a statistical test comparing the Eta squared value (maximum 
possible relationship^ allowing for curvallnearity) and the 
squared Pearson r (assuming linearity) proved to be Insignificant 
at even the *05 level (F = 1*67; d*f* = 3,809; p greater than *10)* 

Table 13 depicts the relationship between item 72 from 
Part IV of the HQ and ratings of instruction* Item 72 provides 
us with an index of rating leniency since It requires the student 
to generate a general rating of instruction and courses at S*I*U* 
The degree to which this item is related to the ratings of any 



70 



TABLE 11 



The Relationship between Percent of Assigned Readings 
Completed and Ratings of Instruction 



Rating Scale 



Percent of Assigned Readings Completed 



Proportion of 
Variance in 
Rating Accounted 

25% or less 25 to 50% 50 to 80% 80 to 90% 90%f for by Percent Uni- 
n = 8X n = 72 n = X56 n « X60 n=342 of Assigned variate 
Readings F-test 



Instructor EvalMation 



3.80 



3.66 



3.75 



3.63 



3.85 



.015 



3.09* 



Course Evaluation 



3.32 



3.32 



3.35 



3.28 



3.49 



.021 



4.23** 



Course Difficulty 



*3 



2.75 



2.89 



2. 75 



2.71 



2.47 



.033 



6.92*** 



* p < ,05 
** p < .01 
*** p < .001 



* p < ,05 
** PC tOl 



TABLE 12 



The Relationship Between the Number of Hours 
Outride of Class Spent Studying^ Reading^ or Working 
for Course^ and Ratings of Instruction 



RatinR Scale 


Less than 1 
n = 79 


1 to 4 

n=369 


Number of Hours Spent Outside of 
Class for Course 

4 to 8 8 to 12 More than 12 
n=250 n=88 n='25 


Proportion of 
Variance in 
Rating Accounted 
for by Hours of 
Outside Work 


Univariate 
F-test 


Instructor 
Evaluation 


3.52 


3.78 


3.84 


3.75 


3.84 


.013 


2.74* 


Course 
Evaluat ion 


3.20 


3.38 


3.30 


3.39 


3.48 


,015 


3,15* 


Course 
Difficulty 
High Score = 
Difficult 


2,71 


2,54 


2.65 


2,88 


2,80 


. ,017 


3.59** 



*3 
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TABLE 13 



The Relationship between the Responses 
to Item 72 from the HQ to Ratings of Instruction 



Item 72 



Response Options 



Rating Scale 
Instructor Evaluation Course Course 

Evaluation Difficulty 



In general, I think instruction 
and classes at S.I.U. are good. 



*3 

CO 



A, Strongly Agree 

n » 96 

B, Agree 

n - 365 

C, Neither agree nor disagree 

n « 214 

D, Disagree 

n = 104 

E, Strongly Disagree 

n =■ 32 

Proportion of Variance in Rating 
Accounted for by the Choice of 
Response Options on Item 72 



Univariate F-test 



4.14 
3,76 
3,74 
3,68 
3,53 

,043 
9,15*** 



3,74 
3,38 
3,41 
3,20 
3,07 

,064 



2,42 
2,68 
2,58 
2,60 

2.43 

,018 



13,89*** 3,67** 



** p < .01 
*** p ^ ,001 
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particular Instructor suggests the degree to which a positive 
or negative rating tendency is biasing an individual student's 
rating of a particular instructor or course. As can be seen 
in Table 13> item 72 accounted for approximately five percent 
of the variance in student ratings of instructor and course* 
Inr section of the mean rating values for each of the response 
options in item 72 suggested that students who strongly agree 
with item 72 tended to rate instructors and courses much higher 
than students vho chose the other response options. 

Since many of the student characteristics examined in this 
study were correlated, an item-by-item analysis can be misleading. 
Vfhen certain factors (student characteristics) are correlated^ 
their relationships with other variables (evaluations) can be 
confounded* Under this condition, a given characteristic might 
be associated with student ratings simply because it ifi highly 
related to another characteristic which is associated with 
ratings^ and not because it makes any unique contribution to 
that variable. In order to partial out the covariance between 
characteristics^ they were simultaneously^ included in prediction 
equations and removed individually to determine the degree to 
which they were uniquely associated with the various rating criteria. 
The results of these analyses are presented in Tables 14 through 
16. Included in Tables 14 through 16 are the zero order correlation 
coefficients relating each item from Part IV of the HQ to the 
rating criterion^ the standardized partial beta weight derived 
from the multiple regression analysis^ and the probability level 
associated with the F*test for each variable as it was removed 
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from the regression equation* When the correlation coefficient 
was computed for a group membership variable (e,g*> sex) > 
a point biserial r Is presented* Also included in Tables 14 
through 16 are the multiple correlations and squared multiple 
correlations between the weighted composite score for the 
set of predictors and each of the criteria* 

Table 14 presents the results of the multiple regression 
analysis for student ratings of instructors* Only nine percent 
of the variance in ratings of instruction is accounted for by 
the student characteristics studied* Two variables made sub*' 
stantlal contributions to the nine percent of accounted for 
variance* They were the general rating of instruction and 
courses at S.I<U< and the grade expected by the student* 
Student GPA was founi to act as a suppressor variable in predicting 
ratings of instruction^ that Is: it correlated zero with ratings 
of instruction^ but aided in prediction by being correlated with 
the error variance left by one of the other predictors* 

In Table 15> the results of the multiple regression analyses 
for ratings of the course are presented. Fourteen percent of 
the variance in student evaluations of the course was accounted 
for by the student characteristics that were examined* As with 
the evaluation jf instructors^ the general rating of instruction 
at S.I.U<> and the expected grade in the course were the best 
predictors* Student CPA again acted as a suppressor variable in 
predicting course evaluations* Question 66 (''Why did you lake 
thin coitrfic?'0 was also found to be related to course evaluations* 

Table 16 shows the results of the multiple re<>ression analysis 
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TABLE 14 
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The Correlations and Beta Weights Relating 
Student Responses to Part IV from the HQ to the 
Student's Evaluation of the Instructor 



Variable 



Correlation 
with Evaluation 
of Instructor 



Standardized 
Partial Beta 
Weight 



Probability Level 
for F-test Upon 
Removing Variable 
from Regression 
Equation 



Course Level 

Sex (1 if male, 0 If female) 
Student's Year In School 
GPA 

Expected Grade In Course 
Reported Deserved Grade 
Why Course Was Taken: 

A. G.S. Requirement 

B. Major or Minor Requirement 

C. Elective for Major or Minor 

D. Satisfy Admission Deficiencies 

E. Personal Choice Elective 

Transfer Status: 

A. All Work at Carbondale 

B. All Work at S.I.U. 
(Both Campuses) 

C. Less than 1 year Elsewhere 

D. Less than 2 Years Elsewhere 

E. More than 2 Years Elsewhere 

Percent of Class Attendance 

Percent of Assigned Readings 
Completed 

Hours Outside of Class Spent 
Studying 

Description of Academic Motivation: 

A. Preparing for the Working World 

B. Getting a Liberal Education 

C. Undecided about Goals 

D. Not really Committed to College 

E. Not in School for a Degree 

General Rating of Instruction 
at S.I.U. 



(a) POINT BISERIAL r 
* a p < .05 NS 

•^t>- P < -01 
ERIC P < -001 



.02 

.05 (a) 

.01 

.00 

.20 

.12 



.03 (a) 

.01 (a) 

.01 (a) 

.05 (a) 

.06 (a) 



.02 (a) 

.04 (a) 

.C3 (a) 

.01 (a) 

.01 (a) 

.06 



.05 



.07 



.06 (a) 

.03 (a) 

.04 (a) 

.00 (a) 

.03 (a) 



.18 



.02 
.01 
.04 

-.08 
.27 

-.09 



.00 
.00 
.00 
.04 
.04 



.00 

.03 
.02 
.01 
.01 

.04 



.01 



.07 



.00 
-.04 
..04 

.03 
-.02 



.17 



P > .05 



Squared Multiple 
Correlation 
Coefficient .092 



NS 
NS 
NS 
* 

*** 
NS 



Removed 

Simultaneously: 
NS 



Removed 

Simultaneously: 
NS 



NS 



NS 



NS 



Removed 

Simultaneously: 
NS 



*** 
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Multiple R 



.303 



TABLE 15 



The Correlations and Beta Weights Relating 
the Student's Responses to Part IV from the HQ 
to the Student^s Evaluation of the Course 



Variable 


Correlation 
with Course 
Evaluation 


Standardized 
Partial Beta 


Probability Level of 
F-test Upon Removing 
Variable from Regression 
Equation 


Course Level . 


,08 




.08 


NS 


Sex (1 If male» 0 If female) 






.03 


NS 


Student^s Vear in School 






-.06 


MS 




-.02 










.24 




27 


*** 


Reported Deserved Gr^ide 






-.04 


MS 


Why Course Was Taken: 










A. G.S. Requirement 


-.09 




-.07 


Removed 


U • flS lU L ** X IIUL Ivdj U X L vvilldit 


.01 




- 0? 




f! Kl tf^ f n Ma i nv* Mf nnv* 

W • u t L X 11 naj 4 1 X IIUL 


.04 


(a) 


nn 

• V V 




D. Satisfy Admission Deficiency 


-.06 


(a) 


- 05 




E. Personal Choice Elective 


.08 








Percent of Class Atnendance 


.09 




.04 


HS 


Student^s Transfer Status: 










A. All Work at Carbondale Campus 


-.01 


(a) 


-.01 


Removed 


B. All Work at Both S.I.U. Campuses .02 


(a) 


.02 


Simultaneously: 


C. Less than 1 Vear Elsewhere 


-.05 


(a) 






D. Less than 2 Years Elsewhere 


.03 


(a) 


00 


NS 


E. More than 2 Years Elsewhere 


.00 


(a) 


.02 




Percent of Assigned Readings 










Completed 


.11 




.05 


NS 


Uour£} Outside of Class Spent 










r^udying 


.09 




.08 


NS 



Description of Academic Motivation: 

A. Preparint for the Working World .04 (a) 

B. Getting a Liberal Education .01 (a) 

C. Undecided about Goals -.02 (a) 

D. Mot Rually Committed to College -.08 (a) 
R. Not in School for a Degree -.01 (a) 



.00 
.00 
.00 
-.03 
.00 



Removed 

Simultaneously. 
NS 



General Rating of Instruction at 
S.I.U. 



.21 



.19 



(a) « point blserial t 
NS =f not significant 
* » p ^ .05 
x p c .01 
iv*#f « p ^ ,001 



Squared Multiple 
Correlation ,]40 



Multiple R 



,374 



TABLE 16 
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The Correlations and Beta Weights Relating 
Student Responses to Part IV from the HQ to 

Student's Rating of Course Difficulty 
(High Score on Dlf^culty Scale = Difficult) 



Vr-iable 



Correlation with 
Rating of Course 
Difficulty 



Standardized 
Partial Beta 
Weight 



Probability Level for 
F-test Upon Removing 
Variables from Regression 
Equation 



Course Level 

Sex (1 If male, 0 If female) 
Student's Year in School 
GPA 

Expected Grade in Course 
Reported Deserved Grade 

Why Course Was Taken: 

A. G, S, Requirement 

B. Major or Minor Requirement 

C. Elective for Major or Minor 

D. Satisfy Admission deficiencies 

E. Personal Choice Elective 

Percent of Class Attendance 

Transfer Statute: 

A. All Work at Carbondale 

B. All Work at Both S.I.U. Campuses 

C. Less tiian One Year Elsewhere 

D. Less tlian Two Years Elsewhere 

E. More than Two Years Elsewhere 

Percent of Assigned Readings 
Completed 

Hours Outside of Class Spent 
Studying 

Description of Academic Motivation: 

A. Preparing for Working World 

B. Getting a Liberal Education 

C. Undecided about Goals 

D. Not Renlly Committed to College 
K. Not Studying for a Degree 

General Rating of Instruction 
at S.I, 11. 



-.17 

-.05 (a) 

..W 

-.03 

-.31 

-.27 



.03 (a) 
.10 (a) 
.04 (a) 
.03 (a) 
.15 (a) 



,11 



.04 (a) 

.08 (a) 

.07 (a) 

.10 (a) 

.07 (a) 



16 



.08 



.05 (a) 

.10 (a) 

.04 (a) 

.02 (a) 

.05 (a) 



-.08 



-.08 
-.02 
.00 
.07 
-.25 
-.03 



-.04 
.00 
-.03 
-,07 
-.10 

-.07 



.00 
.04 
.04 
.07 
.04 



,14 



.12 



.06 
-.02 
.03 
.00 
-.04 



-.09 



MS 

NS 

NS 
* 

*** 
NS 



Removed 

Simultaneously: 



Removed 

Simultaneously: 



A** 



Removed 

Simultaneously: 
NS 



(a) = point biserlal r 



** «= 

iff** a 
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p C .05 
p < .01 
p < .001 



Squared Multiple 
CorrelatlCa .186 



Multiple R 
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.435 



for ratings of course difficulty* Approximately 19 percent 
of variance in ratings of course difficulty was accounted for 
by student characteristics indices* The primary contributors 
to the prediction were ^^the percent of assigned readings complet^d^^ 
index and the ^^expected grade^^ index^ both receiving negative 
weighting coefficients, Other student characteristics that 
were significantly related to the criterion of Course Difficulty 
were GPA^ percent of class attendance^ why the course was taken 
(G,S, requirement^ personal choice elective^ etc,)> student^s 
transfer status^ hours outside of class spent studying^ and the 
general rating of instruction and courses at S*1*U, 
Conclusions and Summary 

The most interesting aspect of this study was the fact that 
the majority of the predictors (student characteristics) were 
found to be independent of student evaluations of instruction* 
Since only nine percent of the variation in ratings of the 
instructor was accounted for by the student characteristics 
examined in this study^ 91 percent of the variation is due 
to other sources. Considering the iact that the Instructor 
Evaluation Scale is highly reliable '94), a sizeable 

portion of ''true snore'' variance is left for further study* 
TliU s^amo condition liolds for the Course Evaluation Scale, 

i 

Tho rating of Cour^tc Difficulty was the most predictable scale 
examined, with nineteen percent |f its variation being accounted 
for by the Student characteristics indices* Due to the extremely 
powerful .statistical tests used in the regression analyses, 
only those characteristics which were significant beyond the 
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*001 level of confidence were interpreted* In keeping with this 
restriction, the data suggests that the student characteristic 
which was most directly related to evaluations of instruction 
was the grade the student expected to receive in the course* 
This variable uniquely accounted for three to four percent of the 
variance in student evaluations* For the Instructor Evaluation 
and Course Evaluation Scales, the general rating of courses 
and instruction at S*I*U* was also found to be predictive* The 
percent of readings completed by the student was similarly 
found to be predictive of student rating of course difficulty* 
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APPENDIX 1 



Part IV of the HQ 
Part IV: Research Data: Items 61 through 72 

In order to revise and improve this questionnaire^ we need to ask a few questions 
about you^ the respondent* This data will be used to supplement the norms for 
different course levels and adjust for different student Interests and goals* 

MARK ONLY ONE LETTERED RESPONSE POR EACH ITEM* OMIT ONLY THOSE ITEMS THAT ARE 
CLEARLY INAPPROPRIATE FOR YOU* 

Start with Item Number 61 on the Answer Sheet 



MARK ONLY ONE LETTER 

61. Your sex: 
A, Male 
B * Fentale 

62* Class year: 
A* Freshman 
B « Sophomore 
C* Junior 
D* Senior 
E« Graudate 



63, 



64* 



65* 



66* 



Your SIU Cumulative Grade 

Point Average: 

A* above 4*0 

B, between 3*5 and 3*^ 

C* between 3*0 and 3*4+ 

D* between 2*0 and 2*9+ 

E* 2*0 or below 

Uhat grade do you expect to 
receive In this course: 
A* A 
B* B 

C* C j 
D* D 
E* E 



67* How often did you attend class? 

A* Less than 50 percent of the time 
B* Prom 50 percent ot 80 percent of 
the time 

C* From 80 percent to 90 percent of 
the time 

D* From 90 percent to 95 percent of 

the time 
E* Over 95 percent of the time 

68* Describe your transfer status* 
(Mark only one*) 

A« All college work done at SIU 

(Carbondale) 
B* All college work done at SIU 

(all campuses) 
C« Less than one academic year of work 

completed at another college (s) 
D* From one to two academic years 

completed at another college (s) 
E* More than two academic years 

completed at another college (s) 



69* 



ERIC* 



Uhat grade do you feel you 
deserve In this course? 
A, A 
B* B 
C* C 
D* D 
E* E 

Uhy did you take this course? 
(Mark only one)^ 

A* Fulfill General Studies Requirement 
B* Required for major or minor 
C* Elective f^rv: major or minor 
D* Satisfy adjulsslons deficiencies 

Personal choice elective 



70* 



What percentage of the assigned 
readings did you complete? 
A* Less tjhan 25 percent 
B* From 25 to 50 percent 
C« From 50 to 80 percent 
D* From 80 to 90 percent * 
E* 90 percent or move 

On the average^ how many outslde^of- 
class hours per veek did you spend 
studying^ readings and working for 
this course? 

A* Less than 1 hour per week 
B* From 1 to 4 hours per week 
C* From 4 to 8 hours per week 
D* From 8 to 12 hours per yeek ' 
E* More than 12 hours per yeek 
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71* Which statement describes you best as a student? 

A* Although a liberal education Is important^ 
1 am most Interested In getting a degree 
to prepare for the working world 

B* 1 am hare primarily to gain a liberal 
education 

C* 1 haven't really decided what 1 want to 
do In the world yet; 1 hope college will 
help me find my place* 

D* I'm not really conitiltted to college^ but 
being here Is better than the alternatives* 

E* 1 am not here to complete a degree* 

72* In general^ 1 think Instruction and classes at 
SlU are good* 

A» I strongly agree with this statement* 

1 agree with this statatient* 
C* 1 neither agree nor disagree with this 

statement * 

1 disagree with this statement* 
E, 1 strongly disagree with this statement* 



Student Affairs 
Research and 
Evaluation Center 



Southern Illinois 
University at Carbondale 
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A Description of Effective College 
Teaching In Five Disciplines as 
Measured by Student Ratings 

John T» Pohlmann 

Southern Illinois University^ Carbondale 

Student ratings of teachers In five disciplines (science and math^ 
education^ social sciences, humanities and business) were analysed to 
determine which teacher attributes were Important In predicting ratings 
of teaching effectiveness* Ratings results from 1439 courses taught 
at Southern Illinois University, Carbondale from 1973 to 1974 were used 
as data for this study* The results Indicated that the Instrudtor 
attributes rated as characteristic of effective Instruction Were highly 
consistent across disciplines, and the effective Instructor was described 
as, 1* Knowing when students understood him, 2* Increasing students 
appreciation of the subject matter, 3* Answering Impromptu questions 
satisfactorily, 4* Achieving the objectives of the course, and 5* Giving 
several examples to explain complex topics* 
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A Description of Effective College 
Teaching in Five Disciplines as 
Measured by Student Ratings 



John T. Pohlmann 



Southern Illinois University, Carbondale 



Student ratings are one of the most frequently used means of evalua-^ 
ting college instructors* In many institutions results obtained from stu^ 
dent ratings affect decisions regarding pay raises, retention and promotion* 
If student ratings are to be used for such important decisions, faculty 
members should be appraised of the teaching characteristics which contri- 
bute to favorable student evaluations. Further, if there are discipline 
differences on what constitutes effective teaching using student ratings, 
this would have implications for the development of discipline specific 
rating forms . 

The purposes of this study were 1. to describe the differences in 
teacher attributes in five academic disciplines, 2» to identify the charac- 
teristics of teachers which account for variation in a general rating of 
teaching effectiveness, and 3* to determine if the teacher characteristics 
that contribute to high student ratings differ among five disciplines 
(science, and mathematics, education, humanities, social sciences^ and 



to assess the criteria used by students in their evaluations of faculty. 
The attributes of teachers which were most comiQonly .:)<^ntioned by students 
as evidence of excellent teaching were preparedness, clarity, and stimula*** 





In their recent review of the literature on student ratings^ Cost in. 



Greenough, and Menges (1972) summarized a number of studied which attempted 
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tion of students* intellectual curiosity (Costin, elait, 1972, 530), 
Deshpande, VJebb and Marks (1970), in a study of student perceptions of 
engineering instructors, found that the effective engineering instructor 
received high student ratings on motivation, structure, content m.^stery 
and instruction skill, Isaacson, McKeachie, and Milholland (1963) related 
selected personality characteristics, as assessed by instructor self- 
reports and peer group nominations, to student ratings of the 'overall 
ability'' of teaching fellows in introductory psychology. These authors 
found that the effective teaching fellow possessed a personality structure 
x^hich was described as artistic, polished, effectively intelligent and 
imaginative^ More recently, McKeachie, Lin and Mann (1971) reported the 
results of a number of studies which examined the relationship between 
teacher warmth and effective teaching, as assessed by student achievement 
residualized for academic ability. Mixed results were obtained. In some 
courses, teacher warmth correlated positively with student achievement, 
while in other courses the relationship was negative. Turner (1970) upon 
reviewing the mixed results obtained in the McKeachie^ ^ ai* (1971) study, 
and other studies, concluded that contextual variables, course type, student 
sex, etc, are potent factors in determining which instructor characteristics 

J ' 1 I 

will prove to be effective, j 

Another group of studies which attempted to Identify the characteristics 
of the effective teacher used simple student descriptions of the effective 
teache" , Downie (1952), in a survey of 16,000 college students, found that 
the attributes of a teacher that were listed as important were; (1) com- 
prehensive knowledge of subject matter, (2) Interest in the subject, (3) 
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bolng proparod for class, and (4) motivating students to do thoir best* 
Crawford and Bradshaw (1968) subjected a number of teacher characteris- 
tics to a paired-comparison scaling analysis by various groups (student, 
administrators, and teachers) and those characteristics which obtained the 
highest scale values in terms of being essential for ^'effective University 
teaching*' were: (I) a thorough knowledge of subject matter^ (2) giving 
well-planned and organized lectures, (3) enthusiasm and interest In teach* 
ing, and (4) a student orientation and willingness to assist outside of the 
classroom. Cadzella (1968) asked a group of students to list criteria they 
would use for selecting the ideal professort The four most Important cri"* 
teria selected were: (I) knowledge of subject niatter, (2) interest in the 
subject, (3) flexibility, and (4) preparation. Costin (1968) had over 200 
students rate the frequency of occurrence of various classroom behaviors 
eJchibited by the %ast lecturer" they had ever had. The attributes that 
received the highest ratings of frequency of occurrence were: (I) acted 
interested in the material, (2) was well prepared, (3) used relevant exam- 
ples, (4) followed a logical sequence of thought^ and (5) explained clearly. 

This series of studies suggested that college students equate effec- 
tive teaching with three broad clusters of instructor attributes, knowledge 
of subject matter, organization of that subject matter for a cleaxj and logi* 
cal preseiitation, and a demoTistration of an interest in the subject matter* 
These clusters indicate a strong subject matter orientation of students in 
the selection of effective college instructors. 

METHOD 

The Rating Fori^ 

The rating form used in this study was the Instructional Improvement 
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DoGstlonnalre (HQ) (Pohlmann, 1973)» The HQ Is a questionnaire designed 
to collect student evaluations of Instructors and courses* Approximately 
30,000 student evaluations contributed to the results* The students ros-* 
ponded to the HQ Items using a 5^olnt scale (5='exceptlonal performance,*** 
l=-lmproveinGnt definitely needed)* For the purposes of this study, only those 
Items relating to Instructor performance were analy;sed* The HQ Items used 
In this study are presented In Table 1* 
'* 

Data 

The data for this study consisted of the results obtained on the HQ 
for 1*439 courses at Southern Illinois University^ Carbondale In 1973 and 
1974* These courses came from virtually every department on campus and every 
course level* The results for each course^ and the results used In these 
analyses, consisted of item means on each of the Items* Consequently^ only 
between course rating variation was analyzed* 

Each of the 1,439 courses that contributed data for this study were 
classified Into five disciplines, 1* Science and Mathematics, 2* Education, 
3* Social Sciences, 4* Humanities^ and 5* Business* 

The departments that were classified Into each of the disciplines 
and the number of courses for each discipline were as fo31o\7i!: 
I I* Science and Mathematics (!J=349) 



Biological Scle^ices 
Botony 
Chei^lstry 
Geology 
^ Matheniatics 
Physics 
Zoology 

Computer Science 



II* Education (K=157) 



Elementary Education 
Secondary Education 
Special Education 
Physical Education 
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Health Education 

III* Social Sciences (N^596) 

Anthropology 

Economics 

History 

Psychology 

Sociology 

Political Science 

1V» Humanities (K=249) 

Dance 

Language Arts 
^Juslc 

Philosophy 

Speech 

Spanish 

French 

English 

V. Business (N=Sfl) 

Accounting 

Adnilnlstratlon 

Business Administration 

Finance 

Marketing 

Statistical Analyses 

The statistical analysis was conducted In two phases* The first phase 
compared the disciplines according to those attributes which received the 
highest and lot:est evaluations* The second phase of the analysis compared 
the disciplines according to what teacher attributes the students In aach 
discipline felt were Important for effective teaching* The following section 
presents a step by step description o£ the analyses: 

Phase 1 

Step 1 . The Item means on the HQ for each course were converted 
to normative T"Scores» Each Item mean was transformed to a scale 
where the unlversity^-wlde normative course mean was 50 and the 
standard deviation was 10* 

Step 2 > The T-scores obtained in Step 1 were then averaged for each 
discipline * 
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Stgp_3_* The Items were then ranked In each discipline to deter- 
mine the teacher attributes that vere rated highest and lowest 
In each discipline. 

Step 4 . The five disciplines were then Intercorrelated over their 
Item means. The elements In the data matrix were the mean T-scores 
on each Item for each discipline. The rows of the data matrix were 
the T-score means for each of the 21 HQ items. The columns of the 
data matrix were the T-score means on the 21 Items for each disci- 
pline. This 21x5 (Item means by disciplines) matrix was then Inter- 
correlated by columns. The resulting R matrix was then Interpreted 
as a discipline similarity matrix based on the discipline profiles 
across rating items. 

Phase 2 

Step 1 . The Item means on Items 1-20 were correlated with Item 
21» the general rating Item. If an Item correlated highly with 
Item 2l» It was assumed that the teacher attribute assessed by 
that Item was a good discriminating attribute to distinguish bet- 
ween effective and Ineffective teaching from the students* perspec- 
tive. 

Step 2 . The Items from the HQ were ranked according to their cor- 
relation with the general rating Item (Item 21) In order to describe 
the Important and unimportant teacher attributes for each discipline. 

Step 3 . The disciplines were then correlated over the Item cor- 
relations with Item 21. The dements of the data matrix were the 
correlations between Item means on Items 1-20 of the HQ and Item 
21. The rows of the data matrix were the Items 1-20, and the col- 
umns of the data matrix vere the 5 disciplines. This 20x5 data 
matrix was then processed to obtal^x a 5x5 R matrix of discipline 
similarity coefficients based on student perceptions of what con- 
stituted effective Instruction. 



RESULTS 

The standardized rating scores on each of the HQ rating Items, for 
each discipline, are presented In Table I. Ikimanlties courses received the 
highest ratings, followed in order by Education, Social Sciences, Business 
and Science and Mathf^matics . 

Within each discipline. Instructors tended to generate different 
rating profiles. Science and mathematics Instructors received their 
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highest ratings on 1. promptly returning homework and tests, 2. being 
dependable In holding class as scheduled, and 3. making clear assignments. 
Science and mathematics Instructors received their lowest ratings on 1. 
encouraging student participation, 2. speaking understandably, and 3. 
knowing IE students understood them. Education Instructors received 
their highest ratings on i. specifying the objectives of the course, 
2. encouraging student participation, and 3. achieving the objectives 
oE the course. Education Instructors received their Icwest ratings on 

1. making clear assignments, 2. grading Ealrly, and 3. being prepared Eor 
class. Social Science courses received their highest ratings on 1. giving 
several examples to explain complex topics, 2. being prepared Eor class, 
and 3. showing an Interest In tbe course. Social Science courses re" 
ceived their lowest ratings on 1. speclEylng the objectives oE the course, 

2. achieving the objectives oE the course, and 3. showing an interest In 
students. Humanities Instructors received their highest ratings on 1. 
answering impromptu questions satis Eactorlly, 2. being prepared Eor 
class, 3. making clear assignments and 4. encouraging student partici- 
pation. Hjmanitles instructors received their lowest ratings on 1. 
speclEylng objectives oE the course, 2. promptly returning homework and 
tests, and 3. being available outside oE class. Business Isntructors 
received their highest ratings on 1. making clear assignments, 2. being 
dependable in holding class <is scheduled, 3. speaking understandably, and 
4. promptly returning homework and tests. Business instructors received 
their lowest ratings on 1. showing an interest In students, 2. achieving 
the objectives oE the course, and 3. increasing students' appreciation 

oE the subject . 



99 



9 



In the next analyslsi the disciplines were Intercorrelated over their 
mean ratings shown In Table 1* The resulting correlations may be viewed 
as profile similarity coefficients* A high correlation between two dis- 
ciplines would suggest a similar profile of mean ratings over the 21 rating 
ltems> and conversely a low correlation would Imply a dissimilar profile 
of mean ratings over Items* The results of this profile analysis Is pre- 
sented In Table 2* 

The correlations In Table 2 Indicate that Education and Social Science 
Instructors are the most dissimilar* Education Instructors tended to re- 
ceive high ratings on Items where Social Science Instructors received low 
ratings* A significant (0^= *05) negative correlation was also observed 
between Education and Science and Mathematics* A significant positive 
correlation was observed between Science and Ifathematlcs Instructors, and 
Business Instructors, suggesting that a common set of strong and weatc 
attributes was exhibited by Instructors In those disciplines* 

The next analysis consisted of correlating the mean ratings on Items 
I thru 20 with the mean rating on the general rating Item (^f2I), "In 
general, the Instructor taught the class effectively*" This analysis was 
conducted separately for each discipline* If an Item correlated highly 
with Item 2I> the attribute assessed by that Item was assumed to be an 
Important teaching attribute* This analysis Is presented In Table 3* 

The correlations In Table 3 Indicate that the great majority of the 
Items on the HQ correlated highly with Item 21* Previous factor analyses 
of the HQ (Pohlmann> 1973) revealed a strong general factor running through- 
out the questionnaire! so the high correlations In Table 3 were not too 
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surprising* \fliile the correlations in Table 3 tended to be high, they 
were not uniformly high, and the variation in the correlations was of 
primary concern in this study. 

For the entire sample, the items which made strong contributions 
to item 21 were items 5* "Knew if students understood him*", 11, "In- 
creased students* appreciation of the subject matter*'*, 7, **Answered 
impromptu questions satisfactorily***, H, *'Achieved the specified ob- 
jectives of the course*", and 9, "Gave several examples to explain 
complex topics*'* 

The results in Table 3 also revealed a highly consistent pattern of 
correlations across the five disciplines* This implied that the students 
in the various disciplines tended to agree on the attributes that were 
indicative of effective teaching* In order to examine the inter-discipline 
similarity issue further, the disciplines were intercorrelated over the 
item correlations presented in Table 3* The esulting R matrix is pre- 
sented in Table A* The correlations in Table 4 further demonstrate the 
strong agreement among students taking courses in the various disciplines* 
Students taking courses in thu five disciplines examined in this study did 
not differ materially in their opinions of x^at teacher attributes charac- 
terize effective teaching. 

DISCUSSION 

The purposes of this study were 1* to describe differences in 
instructor attributes in five disciplines, 2* to identify those instruct 
tor characteristics which account for variation in general student 
rating of teaching effectiveness, and 3* to determine if the teacher 
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characteristics that contribute to high student ratings differ among 
five disciplines* 

The results relating to the first purpose mentioned above indicated 
that different teaching styles are exhibited by instructors in the five 
disciplines examined in this study* Science^ Humanities and Business 
instructors received the highest ratings on making clear assignments and 
being prepared for class* Social Science instructors obtained high 
ratings on knowledge of subject matter and giving several examples to 
explain complex topicst Education instructors received their highest 
ratings on specifying the obj^.ctives of the course and increasing students* 
appreciation of the subject matter* 

The students' who participated in this study rated highly those 
inf;tructors who 1. knew when students understood them> 2* increased 
students^ appreciation of the subject matter^ 3* answered impromptu 
questions satisf actorily> 4, achieved the specified objectives of the 
course > and 5> gave several examples to explain complex ideas* In general^ 
students rated instructors favorably if they were perceived as effective 
in communicating subject matter to students> and communicating the subject 
in a way that helped stimulate student interest in the material* 

There were no substantial differences among students in the five 
disciplines in their perceptions of the importance of teacher attributes 
that characterized teaching effectiveness * The inter-disclpline similarity 
coefficients reported in Table 4 reveal highly consistent patterns in the 
importance attached to the various teacher attributes by students* Students^ 
regardless of their discipline^ tend to agree on the teacher attributes 

i 02 , 
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indicative of effective instruction. 

These results suggested that there are teacher attributes which are 
consistently perceived as Important by students, regardless of their dis- 
ciplines, while teaching styles differed among the disciplines. It Mas £]so 
apparent that those disciplines that received the highest ratings on th^ 
items students perceived as important also received the highest overall 
student ratings. Consequently students tended to reward, with good ratings, 
those teachers who exhibited the attributes students perceived as impor- 
tant determiners of effective instruction. 

These results also have implications for universities who rely on 
student ratings to evaluate their instructors. This study found consider- 
able differences among five disciplines on both the general level of ratings 
received (Table 1), and the profiles of mean ratings (Table 2). It would 
therefore behoove institutions to allow for discipline differences in the 
use and interpretation of student rating results. Allowances for discipline 
differences could be made by comparing faculty members only with their 
discipline peers or developing discipline specific student rating instru- 
ments. 

Student ratings, lilce any assessment tool, can provide very meaningful 
information only if they are properly interpreted. This study indicates 
that a proper interpretation of student rating results must include allow- 
ances for expected discipline differences. 
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Table 1 



Mean Standard Scores (T-scores) Obtained on 21 HQ Items In Five Disciplines * 
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52.6 


43.8 


52 .S 


47.7 


17. 


Knew his subject matter 


AS .9 


50.7 


50.7 


53.3 


50.2 


n. 


Was available outside of class 


A8.3 


50.9 


49.9 


52.7 


5f).l 


10. 


Encouraged student participation 


.6 


54.1 


*4S.7 


54.0 


40.4 


20. 


The course was well organized 


A8.0 


52.7 


49.6 


52.9 


49.8 


21. 


In general^ taught the class effectively 


A7.7 


52.0 


49.2 


S3. 9 


49.1 




Mean Rating for all items 


A7.9 


52.0 


49.4 


53.3 


49.2 



* A high mean connotes a favorable evaluat^.on. 
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Table 2 

The Pearson Correlations Among Disciplines 
Based on Mean Ratings Across 
21 HQ Items (K=21) 



Discipline 2 3 J 5 

1. sci.nce and -56* _ 06 

2» Education ^5 ,21 



3* Social Sciences 
/i » ITunanitics 
5* Business 



A r significantly different from 0,o<.'=*05, two^tailed 
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Table 3 



Correlations between Item 21^ "In general^ the Instructor taught the class at 
fectlvely'^ and Items 1 through 20 from the HQ for Viva Academic Disciplines 

Science and Social 
Item Math Education Sciences Humanities Business 







1=349 


N=157 




!=596 


R=249 




N=88 




r 


Hank 


r 


Hank 


r 


Hank 


r 


Rank 


r 


Hank 


1 


.75 


7.5 


.81 


11.5 


.81 


6 


.79 


11 


.79 


6 


2, 


.64 


16 


.81 


11.5 


.76 


11 


.75 


14.5 


.66 


15 


3 


.60 


17 


.62 


18 


.64 


17 


.63 


19 


.49 


19 




.73 


9 


.71 


17 


.69 


8.5 


.74 


16 


.68 


14 


5 


.85 


.1 


.91 


1 


.84 


3.5 


.86 


3.5 


.89 


1.5 


6 


.68 


13.5 


.76 


13 


.76 


11 


.83 


6 


.78 


7.5 


7 


.82 


3 


.89 


3 


.84 


3.5 


.88 


1 


.84 


5 


8 


.72 


11 


.33 


9 


.79 


8.5 


.82 


7.5 


.74 


10 


9 


.82 


3 


.36 


4 


.81 


6 


.78 


12 


.85 


4 


10 


.72 


3 


.84 


6.5 


.73 


13 


.82 


7.5 


.69 


12 


11 


.82 


3 


.90 


2 


.87 


1 


.87 


2 


.86 


3 


12 


.55 


19 


.46 


20 


.52 


19 


.56 


20 


.51 


18 


13 


.68 


13.5 


.72 


16 


.79 


8.5 


.69 


17 


.69 


12 


U 


.79 


5 


.84 


6.5 


.36 


2 


.86 


3.5 


.89 


1.5 


15 


.42 


20 


.55 


19 


.48 


20 


.64 


18 


.45 


20 


16 


.78 


6 


.84 


6.5 


.76 


11 


.34 


5 


.76 


9 


17 


.67 


15 


.75 


14 


.71 


14 


.81 


9 


.69 


12 


13 


.57 


18 


.73 


15 


.61 


18 


.75 


14.5 


.61 


16 


19 


.75 


7.5 


.84 


6.5 


.67 


16 


.80 


10 


.59 


;7 


20 


.72 


11 


.82 


10 


.81 


6 


.77 


13 


.78 


7.5 
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Table A 

The Pearson Correlations Among Disciplines 
Based on Correlations Between HQ Item 21 and 
the Other HQ Items Appearing in Table 3 
(K=20) 



Discipline 
!• Science and Hath 
2. Education 
3> Social Sciences 
A. Humanities 
5> Business 



2 

.86* 



3 

.88 
.86 



4 

.79 
.92 
.78 



5 

.87 
.83 
.93 
.83 



* All correlations were significantly different from 0, OC =.05, two-tailed, 
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EVALUATING INSTRUCTIONAL EFFECTIVENESS 
WITH THE INSTRUCTIONAL IMPROVEMENT QUESTIONNAIRE 



Dr* John T. Pohljnann 



Counseling and Testing Center 
Southern Illinois University, Carbondale 



no 



One of the most perplexing problems facing instructors and administrators 
is the evaluation of an instructor's teaching effectiveness. All would agree 
that any evaluation method devised for this purpose should possess characteristics 
that will insure fairness and objectivity. A system of evaluation should be 
objective in that the results derived from it will be (1) free of administrator's 
subjective biases, (2) based on concrete observations, and (3) a comtnon system 
applied to all faculty members equally. Further, the information obtained 
from the evaluation should provide constructive suggestions for the improvement 
of instruction^ 

Student ratings of instructors are perhaps the most researched, and most 
reliable means for evaluating instructors. The research history of student 
evaluations extends back into the 1920*5. In sum, the research evidence 
amassed on student ratings indicates that they are a very reliable index of 
instructor excellence. However, regardless of how reliable student ratings 
may be, they provide only one of many possible means for evaluating instructors. 
The ideal evaluation system should contain numerous measures of instructor 
effectiveness such as peer evaluation, administrator evaluations, and measures 
of student learning, in addition to student ratings. 

Student ratings should be included in the evaluation of instruction for 
a number of reasons. First, students have ample opportunity to observe an 
instructor in the act of teaching. Hence, their evaluations are based upon 
first hand experience. Rarely do an instructor's peers or chairman have time 
to spend many hours observing the Instructor. Usually colleague^s impressions of 
an instructor's teaching ability are based upon second hand information, much 
of which is informal student comment. Secondly, when ratings are based upon 
many raters (students) they are much more reliable than ratings based upon 
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a sample of one colleague or chairman* Thirdly^ students are the best 
judges of certain aspects of a course* For example; students are the best 
judges of vhether they understand material presented In lectures or the text* 
They are also the best ones to determine 1£ prerequisite courses have prepared 
them to receive nev material* 

In response to the need for timely and reliable student feedback on 
Instructor and course effectiveness^ the Instructional Improvement Questionnaire 
(HQ) \tas developed at Southern Illinois University's Counseling and Testing 
Center* The remaining sections of this document describe the IIQ^ the computer 
report accompanying Its use^ and the research that has been conducted vlth 
the HQ, 

The Instructional Improvement Questionnaire (HQ) 

The HQ Is a questionnaire designed to collect evaluative feedback from 
students on their Instructors and courses* A copy of the HQ Is provided In 
the Appendix for your Inspection* There are four parts to the IIQj (1) a class 
characteristics section^ (2) an Instructor evaluation section^ (3) a course 
evaluation section and (A) an optional It^ section* These sections are 
numbered on the copy o£ the HQ provided In the Appendix* 

Vnc class characteristics section collects Information from the student 
on variables that have been related to student ratings of Instructors^ or are 
of administrative Interest* The data obtained In this section consists o£ 

(1) student grade point average 

(2) the grade expected by the student In the course he Is 
evaluating 

(3) the ntjmber of hours per treek spent studying for the course 
(A) student class level 

(5) vhether the course Is required or an elective 

(6) student sex 

(7) student college affiliation 
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Some of the information obtained in this section has ^een shown to 
be related to the ratings given by students* Research has indicated that 
there is a tendency for upper classmen to give better ratings than lower 
classmen^ and elective courses tend to get better ratings than required 
courses* When information on these variables is available it is possible to 
moderate evaluative decisions on the basis of such information* For example 
poor ratings obtained by an instructor teaching a required freshman level 
course, would not weigh as negatively, as the same rating obtained by an 
instructor teaching an elective graduate level course* The information 
obtained in the class characteristics section allows for special consideration 
of such factors that may tend to bias the evaluations offered by students* 

The instructor evaluation section consists of twenty items that allow the 
student to rate various aspects of instructor performance* On each item the 
student is asked to rate his instructor on a five-point scale, ranging from 
"exceptional performance" to ^^improvement definitely needed*^' 

The course evaluation section consists of twenty statements about various 
aspects of the course* Some statements are phrased positively and some are 
phrased negatively* The students respond to each statement on a five^point 
scale ranging from "strongly agree" to "strongly disagree*" A good rating 
is obtained if students agree with positive statements, or disagree with 
negative statements* 

The optional item section consists of sixty reponse positions that 
the instructor may use to record student responses to locally supplied items* 
It is virtually impossible to design one questionnaire to satisfy everyone's 
needs for student feedback* There are many specific aspects of a course that 
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may be evaluated by supplying students with optional itemi;. Specific 
chapters in a book, novel teaching strategies, handouts and class projects 
may all be evaluated in this section. All that is required is for the 
Instructor to supply his students with a list of items numbered anywhere 
from 41 to 100, and Instruct the students to respond to those items in the 
optional Item section. The only restriction on the optional questions is 
that the number of response options must be no more than five. 
Computer Analysis of the IIQ Results 

The completed IIQ^s are processed in two phases* During the first 
processing phase, the students' answer sheets are scanned on the SIU Testing 
Center's OpScan 100 DM optical scanner and a coded linage of each students' 
responses is written on a magnetic tape* This tape is then used as input 
to the computer during the second phase of processing. A computer program 
developed at the Testing Center takes the magnetic tape prepared by the 
scanning machine, analyzes the data residing on the tape, and produces 
a printed report of the results. A copy of this printed report may be 
found In the Appendix. There are four standard sections and an optional 
fifth section in the printed report. 

Part I; Instructor Evaluationt The first page of the computer printout 
reports the results of the instructor evaluation part of the HQ* A heading 
is printed across the top of the page which contains course and instructor 
Identifying information. The number of students who evaluated the course 
is also presented in the heading* 

The report then provides a listing of the item statements, the percent 
of students who marked each of the response options, an item mean, and an 
item decile* 
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The item mean is determined by assigning weights ranging from 1 to 5 
to each of the response options- A weight of 1 is assigned to the response 
option "Improvement Definitely Needed"^ and a weight of 5 is assigned to the 
response ''Exceptional Performance*" The item mean is simply the average of 
these weights for an item* The item means may range from 1 to 5» and a 
high mean implies a good rating* 

The decile values provided at the right of thii item mean reflect the 
relative location of a particular item mean among the means for a normative 
group of 2,363 courses at SIU* The deciles, range from 0 to 9, with 9 being 
the best decile attainable* If the decile value is multiplied by 10, that 
product will indicate the approximate percentage of courses in the normative 
sample that fell below one's item mean* For example: if the instructor 
received a decile value of 7 for an item, that would indicate that approximately 
70% of the normative group fell below him or, his ii:em mean placed him in the 
ipper 307. of the normative group on the attribute asi;essed by that item* 

Part II; Course Evaluation * The second page of th^ printout contains the 
results for the course evaluation section of the HQ* The item statements, 
response percentages, item means, and deciles are given for each item* Some 
of the items in tikis section are phrased negatively (e*g*, item 31); consequently 
the scoring is reversed on these items so that disagreement leads to a high 
mean* A high mean implies a good rating for all items* Decile values are 
also provided for each item* 

Analysis by Subscores * The third page of the report presents the results 
of the evaluation by subscores* The subscores are average item means for 
groups of items that have similar content* For example: the Course Difficulty 
subscore is the average of the item means for items 25,36,38 and 39* Each 



ERLC 



115 



-6- 



of these items is concerned with course attributes related to difficulty* 
By combining these items into a common score, that score serves as a more 
reliable index of course difficulty than any one of the original items* The 
subscores also allow for summarizing the results of the evaluation^ It is 
much easier to interpret five subscores than 40 item means* However, analysis 
of the item by item results is essential if an instructor wants to get the 
best picture of his course* 

A norm table is provided for each subscore, so that an instructor can 
determine the relative location of his subscores in different norm groups* 
The decile value of each subscore are presented for norm groups of required 
and elective courses at five course levels* These norm tables allow an instructor 
to compare his results with other courses similar to his in terms of course 
level and elective status^ This comparison is fairer than comparing all 
instructors against a common standard, since research indicates different 
ratings are expected depending upon course level and whether or not the 
coure was required* < 

The Class Characteristics Section * The next page on the printout Is 
a tabulation of the students responses to the student characteristics information 
obtained on the HQ* This information may prove to be helpful for identifying 
peculiar class characteristics* The CPA information provides an index of the 
academic ability of the students enrolled in the course* The number of 
outside study hours per week provides an index of the work level required 
of students. This information may lead an instructor to increase or decrease 

* 

the course requirements depending upon the number of hours the students report 
studying. Information on variables 3uch as expected grades, year In school 
and college affiliation are also provided in this section* 

^ 1 1 1^ 
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Optlonal Items Section * The last page of the printout is an optional 
section* The results of the students^ responses to any optional Items that 
were Included are presented here* If no optional Items were Included^ 
no results ivlll be printed* The results consist of response percentages and 
Item means* Up to sixty Items may be Included In the optional Item section* 
SuMnary of Research on the HQ 

Research conducted on the HQ has fallen Into three broad categories: 
(1) faculty reactions to the IIQ> (2) studies of the reliability or stability 
of HQ results^ and (3) studies of the relationship between selected class 
characteristics variables and HQ results* 

Faculty Reaction* Any evaluation system^ If It Is to be received 
favorably by those evaluated, must be perceived as providing Information 
which Is accurate, and helpful for making constructive changes In teaching 
behavior* In order to determine If the HQ evaluation possessed these attributes, 
a group of faculty members who had participated In the HQ evaluation program 
were polled, via a questionnaire* One hundred and two useable questionnaires 
were available for analysis. This figure represented a $3% return rate* One of 
the questions on the questionnaire was directed at the helpfulness of the HQ 
results in improving instruction* Five response options were provided, and 
the percent of faculty choosing each option was as follows: 



Response Option 



Percent 



very helpful 
moderately helpful 
somei^at helpful 
not helpful 
waste of time 



25 
43 
27 
3 
1 



These figures indicate that the HQ results are perceived as helpful by a 



majority of the faculty t4io use the service* 
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Another question included In the questionnaire asked the faculty to 
rate the degree to which they felt the results of the HQ evaluation 
represents an accurate assessment of their teaching performance* Four 
response options were available, and the percentage of responses at each 
option was as follows: 



Response Option Percent 

Yes, definitely 13 

Yes, generally 70 

Yes, to a limited extent 15 

No relationship 0 



These results suggest that the faculty members u^ing the HQ feel that the 
results represent generally accurate indications o£ their teaching effectiveness* 
Reliability and Stability of HQ Results 

One of the most inq)ortant attributes of any measuring device, such as 
the HQ, is its reliability* A test is considered reliable if the measurements 
(scores) obtained from It are stable* A test would be unreliable if the 
results obtained by it varied, such that, at one time the results indicated 
that the instructor was good, but at another time, under the same circumstances, 
the results indicated he was poor* One a^ect of the reliability of 
instructor ratings is the stability of results over changes in students, or 
the degree to which different students agree on the effectiveness of the 
instructor* Another aspect of reliability in this context is concerned 
with the consistency of student responses to the questionion the HQ* 
Consistency o£ responding is present when students respond similarly to 
similar items* Consistent responding w'^uld be present when students who 
agreed with the statement, **The instructor was effective", would also agree 
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with the statement, "The instructor taught effectively*". If students agreed 
with both of these statements, it may be said that they were consistent in 
their responses* These two Interpretations of the concept of reliability give 
rise to two methods for assessing the reliability of a test, one method is 
the test-retest method, and the other is the internal consistency method* 
test-retest reliability is established if the results of a test obtained 
at time 1 correlate with the results from the same test at time 2* Internal 
consistency is established if responses to similar items are correlated* 

Reliability studies with HQ suggest that it possesses both test-retest 
and internal consistency reliability* Table 1 presents the results of two 
reliability studies undertaken with the HQ* The results are presented for 
the five subscales of the HQ* 

The reliability coefficient used to assess t^he internal consistency 
of the HQ subscales was Cronbach's alpha* With the exception of the General 
Course Rating Scale, the coefficients were above *8* The test-retest 
coefficients were calculated by correlating the results of 68 instructors 
ratings for two different quarters* The results were taken from courses 
where the instructors were teaching either the same course, or courses in 
the same sequence* The interval between the evaluations was one quarter, or 
approximately three months* 

Relationships between the HQ Subscores and Other Variables * The data 
from which these analyses were derived were obtained from 1,247 courses 
evaluated with the HQ in 1971* The data was analyzed using the course as 
the data unit* The following indices were then computed on each course: 
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TABLE 1 



RELIABILITY COEFFICIEWrS FOR THE HQ SUBSCALES 



Reliability Coefficients 



Subscale 



Internal 
Consistency 
(Cronbach's «< .) 
N-888 instructors 



Three Month 
Test-Retest 
N"68 instructors 



General Course Bating 
Student Orientation 
Course Difficulty 
Assignioents and Grading 
Presentation of Material 



,62 
,85 
,89 
.85 
,93 



.67 
.71 
.69 
.76 
.71 
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Cla$s Size; the number of persons completing the questionnaire 

for the course* 
Percent Male; the percentage of the class that was male. 
Student Class; the average class level of the students enrolled 

in the course (1 = Freshman, 2 + Sophomore,*.., 5 = Graduate 

Student) . 

CPA; the average self-reported GPA of the students in the class. 
Expected Grade.; the average grade expected by students in the 

class. (A=5,B»4, C=3, D=2, Fail=l) . 
Percent Elective; the percentage of the students Caking the 

course as an elective. 
Outside Hours; the average number of hours per week reportedly 

spent by students on studying or preparing for the course. 
General Rating; the average rating given to Instruction at SIU, 

measured on a 5-poiat scale. 

These indices were then correlated with each of the subscale scores from the 
HQ* The results of these analyses are presented in Table 2. 

With the exception of the average grade expected, the correlations of the 
class characteristics variables with the various subscores is low. In general 
the most important variables, in terms of their correlation with ratings, were 
expected grades, and the percent of students taking the course as an elective. 
It was interesting to note that the average number of study hours tended to 
be independent of ratings of the instinictor. This result tends to contradict 
the notion that instructors can "buy" good ratings by offering a course that 
is not too demanding. The general rating of instruction at SIU was found to 
be independent of specific instructor and course ratings. This finding suggests 
that students rate specific courses independently of their attitudes about 
the quality of instruction at an entire institution. 
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TABLE 2 

THE ZERO ORDER CORREIATIONS 
BETWEEN STUDENT RATINGS 
AND CLASS CHARACTERISTICS VARUBLES 
(K-1247) 



STUDEHT RATINGS 





General 






Assign- 


Presen- 


Class Characteristics 


Course 


Student 


Course 


ments £e 


tation of 


Variables 


Rating 


Orientation 


Difficulty* 


Grading 


Material 


Class Size 


-.16 


-,25 


-,11 


-,15 


-.17 


Percent Male 


-.09 


T,08 


-,20 


-,02 


-.09 


Student Class 


,17 


.21 


,10 


,06 


,10 


6PA 


,11 


,18 


.05 


,08 


,12 


Expected Grade 


.42 


,48 


,51 


,33 


,35 


Percent Elective 


,27 


.21 


,21 


,12 


.15 


Outside Hours 


,10 


,06 


-.28 


.07 


,02 


General Rating 


,09 


,01 


,02 


,06 


.03 



^Kigh Scores on the Course Difficulty scale reflect an easy course* 
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PART U limiMICTDR EVALUATION (tTEMS 1 29) 

iteSPONSE OPJiOHS: 

* !^ Repaietfk c'ass_ _ _ 

!!^. Msdecieai sijnri^? ts _^ 

Sd cMi sivjJaids'Jof grading ' _ 

G'adw iauv* _ 

K^iew \\ slii'^ -Ills un^^ef^^ooa hitn_ 

s/gokeundwthiidabj^^^^ _ * _ 

Aitswered iwofliiji/ISioilV^ „^ 

;>wed an ivlereJiri _ _ _ . 

toe sevei'ji cxalpt^^gfplal^^ _ 
Accepied ci.ticisJ^JdsijgKj^ _ _ 
Increased y^^i: ajjgfcciSTW™ (he subif*^t 
Was Jerendobie injiotdint cIbss as scheduled 
S^cilied cbjectim i^Mhe touisc_ 
Achieved lii^ specif lea r.t>n*t5:v» of ihe course 
Proiii0l[)[^Hu'ned honeworK and le^is 
- [i4 ShweJ an i-leresl \& sJudenis ^ _ 

; ! * Knew his sir>i^u»aVci 

^ Was ayajlatHeotilsMJcol class , ^ 

; . *^ ^ Encci;rageu slndenipailrtiDaliof*^ 

(ngei^'fal, vitigfiMhe class effecFiveiy 
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PART H: COURSE EVAtOATION (tTEMS 2140| 

HiSPONSe OniOhS: 

t : ThiS^coms^ J^as^a^f^jJ leaininj enpaience, 

Tbe_conlent of Ihg couise ms good, 

|_ The Gci]fse A.is^vuell organized. 

I had Iroubir i^yini attention in cjtass. _ _ 
TMie shouk b^^iMKM^ prerequisites. 
Berps^wjftayjMento _ _ 

^_This course «s vArtefHl% 

Th'^jmouiil ^V^^wwotfftastpjMO^^ 
This cojiisp Aa^iiM^iff best I tev^Ute 
t 'fh? tests coveLe^ ihc coutse matariat w ejL ^ 
This course wasiwaste of tiiw,_ _ 

* ThelMtboctwaspw'- 

Ajdio-visuals could be uiedrMKijtf^^^ _ 
^ This couise shouM be Uufhl.inmc ^hirway.^ 

I ttthjred thfS nM|»ial in olher^coufses* 

T^ coi^nr rialefi|l was too dilliculL 

J" ii- J ^ i f , .](ll"?_cou!Sc shoijlj continue _b,bt ol(erei_ 
^^±^ ' . T^iJ*S*"t dssigiments hatd to wjtfslsnd, 

3? -i^.i^^.Jl -'-f .Iwisofjenconlirtid. 

3. j A i i7b t ^ Genera lly, the coyrse ((»d . 
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AHILySIS SUBSCORFSt 

THE FOLLOWING SU(*SCO<*ES WEftE OErI V pO BV I/ERIGING I TF'4 '4E IN S F 3^ ITE HS TH IT HIVI? Cf><4«3N OR SI><! LI^ C D^TENT. 

tHt= coottSP — NbR^MTtrvr^FcTTErTeffpH^^Nri^ 

aftOUPS BISEO UPON PPQUlPtO ANf> ELECTIVE COURSES IT THE flVC COURSF LEVELS. FOR EXIHPLEJ I OeCILE OF THREE (31 

TN THE 300 LEVFL-Reoai^>:0 COURSE C(:LL HEAIS THAT VDIW SUBSCALE SCORE PLACFO VOJ IN A G^OUf^t BELOW WHiCHt 30 PE^CE*4r 

OF THE SOBSCOpfiS FOR 300 LEVEL PEOUIrEO CDORSES FELL* SIMILARLY t A OSClLf OF 4 WOULO I**PLY THAT APPROX I<4ATELr 

40 PERCENT OF THE NORHATlVE SA4PLE F£LL BSLOW YOOR SbBSCORE. TH1;'0ECILES R4NGS FROH 0 TO 9* UlTH 9 BEING 

^HE HIGHEST OECILf ATTAINABLE - 
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FOR ADDITIONAL TECHNICAL AND SUPPDRTING 
INFORMATION SEE: 



Elmore, Patricia B. and La Pointe, Karen A.. Effects of Teacher 
Sex and Student Sex on the Evaluation of College Instructors, 
Journal of Educational Psychology , 1974, pp. 386-389. 



Pohlmann, John T. A Multivariate Analysis of Selected Class 
Characteristic and Student Ratings of Instruction, Multi variate 
Behavioral Research , January 1975, pp., 81-92. 



Pohlmann, John T. A Description of Teaching Effectiveness as 
Measured by Student Ratings, Journal of Educational Measurement , 
Spring 1975, Vol. 12, No. 1, pp. 49-54. 



THESE MATERIALS ARE NOT AVAILABLE ON MICROFICHE DUE TO COPYRIGHT 
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